MOVING DOWN 
THE TRACK 


Changing School Practices 
During the Second Year of 
DIPLOMAS NOW 



mdrc 


BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 



Susan Sepanik 
William Corrin 
David Roy 
(MDRC) 


Aracelis Gray 
Felix Fernandez 
Ashley Briggs 
Kathleen K. Wang 
(ICF International) 


k 


May 2015 



Moving Down the Track 

Changing School Practices During the Second Year of 

Diplomas Now 


Susan Sepanik 
William Corrin 
David Roy 
(MDRC) 

Aracelis Gray 
Felix Fernandez 
Ashley Briggs 
Kathleen K. Wang 
(ICF International) 


May 201 5 


mdrc 

BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 



Funding for this report came from the U.S. Department of Education under its Investing in Innova- 
tion (i3) initiative. The i3 grant called for Diplomas Now to expand its whole -school reform 
initiative and for MDRC and ICF International to conduct an independent evaluation of the imple- 
mentation and impacts of the reform. Matching grants were also provided by the PepsiCo Founda- 
tion and Ballmer Family Giving. 

Dissemination of MDRC publications is supported by the following funders that help finance 
MDRC’s public policy outreach and expanding efforts to communicate the results and implica- 
tions of our work to policymakers, practitioners, and others: The Annie E. Casey Foundation, 
Charles and Lynn Schusterman Family Foundation, The Edna McConnell Clark Foundation, Ford 
Foundation, The George Gund Foundation, Daniel and Corinne Goldman, The Harry and Jeanette 
Weinberg Foundation, Inc., The JBP Foundation, The Joyce Foundation, The Kresge Foundation, 
Laura and John Arnold Foundation, Sandler Foundation, and The Starr Foundation. 

In addition, earnings from the MDRC Endowment help sustain our dissemination efforts. Con- 
tributors to the MDRC Endowment include Alcoa Foundation, The Ambrose Monell Foundation, 
Anheuser-Busch Foundation, Bristol-Myers Squibb Foundation, Charles Stewart Mott Founda- 
tion, Ford Foundation, The George Gund Foundation, The Grable Foundation, The Lizabeth and 
Fra nk Newman Charitable Foundation, The New York Times Company Foundation, Jan Nichol- 
son, Paul H. O’Neill Charitable Foundation, John S. Reed, Sandler Foundation, and The Stupski 
Family Fund, as well as other individual contributors. 

The findings and conclusions in this report do not necessarily represent the official positions or 
policies of the funders. 


For information about MDRC and copies of our publications, see our website: www.mdrc.org. 
Copyright © 2015 by MDRC”. All rights reserved. 



Overview 


Too many students in high-poverty, urban communities drop out of high school, and too few gradu- 
ate prepared for college and careers. Three national organizations — Talent Development Second- 
ary, City Year, and Communities In Schools — have formed Diplomas Now in an effort to trans- 
form urban secondary schools so fewer students drop out and more graduate ready for postsecondary 
education and work. Thank s to a validation grant from the U.S. Department of Education’s 2010 
Investing in Innovation (i3) competition and matching funds from private sources, teams from all 
three organizations are implementing the Diplomas Now data-driven, tiered intervention model in 
schools across the nation. The model combines a comprehensive school reform strategy aimed at 
transforming the academic experience of all students with early warning indicators related to attend- 
ance, behavior, and course performance. By identifying students at risk of dropping out and inter- 
vening with targeted support, Diplomas Now attempts to get failing students back on track. 

MDRC and 1CF International are conducting an independent, experimental evaluation of the impact 
and implementation of Diplomas Now. During the 2011-2012 and 2012-2013 school years, 62 sec- 
ondary schools in 1 1 school districts agreed to participate in this study. Thirty-two of these schools 
were randomly assigned to implement the Diplomas Now model while the other 30 schools were 
assigned to a control group, continuing their existing school programs or implementing other reform 
strategies of the districts’ or schools’ choosing. This second report shares second-year implementa- 
tion fidelity findings, looks at the differences in school structure and staff practice between schools 
implementing and not implementing the program, and presents analyses of qualitative data that 
delve deeper into contextual issues and the integration of the program into the schools. Findings 
from this report include: 

• As in the first year of implementation, during the second year schools implemented a majority 
of program components as the model’s designers intended, but still showed room for growth to 
meet ideal implementation. 

• There are differences in organization and services between schools randomly assigned to im- 
plement the model and similar schools randomly assigned to not implement the model, suggest- 
ing that model implementation is changing schools’ practices in ways that align with the goals 
of Diplomas Now. 

• Model implementation was somewhat hindered by factors external to the program such as staff 
turnover and budget cuts, but Diplomas Now staff members were able to support schools de- 
spite these challenges. Program staff members were most successful when they could align the 
program with school goals and gamer school administrator and teacher support. 
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Preface 


Too many young people growing up in U.S. cities are not graduating high school or are gradu- 
ating unprepared for postsecondary education and the workplace. The Diplomas Now model 
aims to address that problem through an ambitious collaboration of three organizations — Tal- 
ent Development Secondary, City Year, and Communities In Schools — offering a promising, 
multifaceted approach to secondary school refonn and student support. The partners are im- 
plementing a complex set of refonns in middle schools and high schools in 1 1 of the largest 
school districts in the country, with the goal of keeping students from dropping out and helping 
more of them graduate high school prepared for college and careers. The model includes or- 
ganizational, instructional, and curricular support to schools as a whole, and also aims to iden- 
tify students falling off the graduation track and to intervene with additional academic and so- 
cial service support. 

This second report for the federal Investing in Innovation (i3) evaluation of Diplomas 
Now presents findings about the second year of implementation, and explores the context sur- 
rounding the implementation of the program and its integration into schools. It discusses how 
closely schools implementing the Diplomas Now model followed the intentions of the model’s 
designers over the first two years, and also explores the differences in programs and services 
between the schools implementing the model and those not implementing it. This analysis of 
service contrast shows whether Diplomas Now is changing school structures and the services 
offered to students. It illustrates how schools are different from what they would have looked 
like without Diplomas Now. When the impact results become available, the analysis of service 
contrast will help reveal what parts of the Diplomas Now model are effective in creating posi- 
tive change for students in the complex environments of public schools. 

The next report will present results regarding the model’s impact on students’ attend- 
ance, behavior, and course performance, three important indicators of future school success and 
graduation. The more Diplomas Now changes school practices, the more likely it will be to 
have an impact on student success. Subsequent reports will explore longer-term outcomes, in- 
cluding ninth-grade completion for students who entered the study in sixth grade and high 
school graduation for students who entered in ninth grade. 


Gordon L. Berlin 
President 
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Executive Summary 


Diplomas Now is a partnership of three national organizations — Talent Development Second- 
ary, City Year, and Communities In Schools — to transform urban secondary schools so fewer 
students drop out and more graduate ready for postsecondary education and work. The Diplo- 
mas Now model is a comprehensive approach to whole-school refomi that includes structural 
changes, instructional materials and curricula, teacher and administrator coaching and support, 
and an early warning system that identifies and targets students falling off the graduation track. 
The model brings additional people into the school to support model implementation and pro- 
vide additional assistance for students. Acting as a representative for the partnership, Johns 
Hopkins University, home to Talent Development Secondary, was awarded a federal Investing 
in Innovation (i3) validation grant in 2010 to support the expansion of Diplomas Now from a 
few schools to more than 30 across more than 10 school districts. The grant funds also support a 
rigorous experimental evaluation of the Diplomas Now model, led by MDRC in partnership 
with ICF International, that explores Diplomas Now’s implementation and impact. 

This report focuses on the second year of Diplomas Now model implementation. It pre- 
sents several key findings: 

• As in the first year of implementation, during the second year schools im- 
plemented a majority of program components with fidelity to the model, but 
still had room for growth to meet ideal implementation. On average, Diplo- 
mas Now schools (DN schools) were most successful in implementing those 
program components that involved adding new services and technology, 
provided directly by Diplomas Now staff members. DN schools have been 
less successful thus far in implementing those program components that re- 
quire schools or districts to modify their curricula or to institute or expand 
their instructional coaching for teachers. 

• Diplomas Now has resulted in differences between the programs of DN 
schools and those of similar schools not assigned to implement the model 
(non-DN schools), suggesting that model implementation is changing school 
practices in ways that align with the goals of Diplomas Now. These differ- 
ences grew from Year 1 to Year 2, in part because non-DN schools reduced 
their services. This suggests Diplomas Now stabilized schools that might 
otherwise have lost services. 

• Although model implementation was somewhat hindered by factors external 
to the program (such as principal turnover and budget cuts), Diplomas Now 
staff members were also able to support schools through these types of tran- 
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sitions. School and program staff members reported that the Diplomas Now 
school-based teams had often been successful by the second year in becom- 
ing part of the fabric of the school. Program staff members suggested they 
were most successful when they could align the program with school goals 
and gamer school administrator and teacher support. 


Background 

Although the national high school graduation rate has increased over the past decade, one in 
five students still do not complete high school in four years. 1 Among low-income students, al- 
most 30 percent fail to graduate on time. 2 Compared with high school graduates, dropouts are 
more likely to live in poverty, earn less money, suffer from poor health, be incarcerated, or be 
dependent on social services. 3 Students who face the most serious barriers to earning their di- 
plomas are in the greatest need of intensive academic, social, and other interventions to make it 
through high school, and most dropouts are concentrated in low-income, urban high schools. 

Research has shown that it is possible to predict a student’s likelihood of dropping out 
of high school using indicators of poor attendance, poor behavior, and course failure measured 
as early as middle school. 4 These findings suggest that programs may have greater success get- 
ting more students to graduation if they intervene with students who are off track as early as 
middle school. Moreover, ninth grade is a critical year, and researchers have shown that im- 
proving student course perfonnance in the ninth grade can lead to substantial improvements in 
graduation rates. 5 

Among those students who do graduate high school, many do not graduate ready for 
college and need to take remedial (developmental education) courses: over 30 percent of col- 
lege undergraduates and over 40 percent of community college students enroll in such cours- 


'Richard J. Mumane, “U.S. High School Graduation Rates: Patterns and Explanations,” Journal of Eco- 
nomic Literature 51, 2 (2013): 370-422. 

2 Robert Balfanz, John M. Bridgeland, Joanna Homig Fox, Jennifer L. DePaoli, Erin S. Ingram, and Mary 
Maushard, Building a Grad Nation: Progress and Challenge in Ending the High School Dropout Epidemic 
(Washington, DC: Civic Enterprises, 2014). 

’Kristin Anderson Moore, Making the Grades: Assessing the Evidence for Integrated Student Supports 
(Bethesda, MD: Child Trends, 2014). 

4 Robert Balfanz, Liz Herzog, and Douglas J. Mac Iver, “Preventing Student Disengagement and Keeping 
Students on the Graduation Path in Urban Middle-Grades Schools: Early Identification and Effective Interven- 
tions,” Educational Psychologist 42, 4 (2007): 223-235. 

5 Melissa Roderick, Thomas Kelley-Kemple, David W. Johnson, and Nicole O. Beechum, Preventable 
Failure: Improvements in Long-Term Outcomes When High Schools Focused on the Ninth Grade Year (Chi- 
cago: University of Chicago, CCSR, 2014). 
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es. 6 Beyond supporting struggling students to graduation, school improvement efforts need to 
ensure that all students participate and succeed in rigorous curricula that prepare them for col- 
lege and careers. 


The Diplomas Now Partnership 

The Diplomas Now partnership works with schools to ensure that students are getting the sup- 
port they need to (1) get to school and to class, (2) behave in ways that facilitate learning, and 
(3) keep up with the lessons being taught. In other words, the pathway to student success in DN 
schools is linked to attendance, behavior, and course perfonnance: the “ABCs” that predict 
whether students graduate or drop out. Diplomas Now is a set of school reform interventions 
designed to be implemented in underfunded urban secondary schools with many students who 
are not performing well academically, in communities struggling with poverty, where many 
students drop out. The Diplomas Now partners collaborate to help schools provide the right ser- 
vices to the right students on time and at the right level of intensity. They do so by offering var- 
ying levels of support for students with different needs: whole-school restructuring and instruc- 
tional reform to strengthen the educational experience of all students, individual support for stu- 
dents showing early signs of falling off track, and case management for students in need of 
deeper interventions. To determine which students need extra support, the model relies on regu- 
lar monitoring of students’ attendance, behavior, and course perfonnance. See Box ES.l for 
more on what each of the Diplomas Now partners contributes to the overall model. 


The Diplomas Now Model 

The elements of the model are classified as “inputs” that the Diplomas Now partners implement 
in schools. Some of these inputs represent substantial interventions on their own, such as im- 
plementing a rigorous curriculum, or setting up a tiered intervention system to identify at-risk 
students and tailor interventions to their specific needs. Diplomas Now integrates these compo- 
nent interventions into a cohesive model focused on ensuring that all students have a path to 
graduation. The inputs are implemented in collaboration with school staff members and align 
with the Four Pillars of Diplomas Now, a characterization of the model used by Diplomas Now 
staff members to help them organize their work. 


6 U.S. Department of Education, National Center for Education Statistics, 2011-12 National Postsecondaiy 
Study Aid Study (NPAS: 12): Profile of Undergraduate Students: 2011-12, NCES 2015-167 (Washington, DC: 
U.S. Department of Education, National Center for Education Statistics, 2014). 
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Box ES.1 


The Diplomas Now Partners and Their Roles 
Talent Development Secondary 

Talent Development Secondary, based at Johns Hopkins University, provides organiza- 
tional, instructional, curricular, and data support to schools intended to help all students 
achieve at high levels. This school-wide effort includes reorganizing students and teach- 
ers into small learning communities, providing professional development and coaching 
to strengthen teacher pedagogy, supplying college- and career-preparatory course con- 
tent, and creating an early warning indicator data system to identify students falling off 
track. City Year and Communities In Schools play leading roles in providing additional 
services for these students. 

City Year 

City Year is an AmeriCorps program through which young adults, ages 18 to 24, partici- 
pate in a year of full-time national service. These “near peers” (given their proximity in 
age to the students) serve as tutors, mentors, and role models, personalizing the school 
experience of the students. In addition, the AmeriCorps members provide after-school 
programs and help teachers by working with students during class time. 

Communities In Schools 

Through a school-based site coordinator, Communities In Schools, a national dropout- 
prevention organization, draws on school and community resources to move the stu- 
dents at highest risk of dropping out back on track to graduation. The site coordinator 
assesses the needs of a student, develops an individual case plan for that student that 
lays out a strategy to address those needs, and connects the student to services aligned 
with the case plan. 


Pillar I: Teacher Teams and Small Learning Communities 

Diplomas Now collaborates with school leaders to organize schools such that small 
groups of teachers work with the same population of students. These small learning communi- 
ties create opportunities for personalization where teams of teachers know the same students 
and can work together to best teach and support them. Students also share the same classes, and 
become known to one another. These teacher teams and small learning communities function 
best when there are opportunities for teachers to collaborate within the daily schedule and when 
they have classes long enough to cover material in depth and keep up the pace of instruction. 
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Pillar II: Curriculum and Instruction with Professional Development 

This pillar is focused on teaching and learning, and on giving teachers the training and 
resources they need to deliver strong lessons. Through professional development and instruc- 
tional coaching, teachers have an opportunity to sharpen their pedagogy. Having curricular ma- 
terials aligned with college- and career-ready standards and accelerated remediation courses for 
struggling students helps ensure all students can meet their potential. 

Pillar III: Tiered Student Supports 

Providing more intensive support for students with greater needs is the core idea of this 
pillar. The tiered intervention model involves implementing an early warning system that draws 
on data on the ABC indicators for individual students. It relies on staff members having regular 
times to meet to review those data and plan interventions for students who are off track. 

Pillar IV: Can-Do Culture and Climate 

School reform is difficult, and school staff members often have much to do when they 
are asked to effect change. Diplomas Now brings at least a dozen staff members to a school to 
help coordinate school transformation, introduce new practices and structures, provide training 
and support, provide additional services to students, and engage with families and community 
organizations. All of these staff members are trained by Diplomas Now before and throughout 
the school year. Providing and organizing resources to assist the school’s staff helps foster a 
culture and climate where it feels possible to improve the school and support students better. 

Implementing the pillars is hypothesized to affect a series of school outcomes (such as 
the school’s climate and communication among stakeholders) and student outcomes (like study 
habits and engagement with school), leading to improvements in students’ attendance, behavior, 
and course performance, which should in turn lead to increased high school graduation rates. 


The National i3 Evaluation of Diplomas Now 

In total, 62 schools (33 middle schools and 29 high schools) from 1 1 large urban school districts 
across the country were recruited to participate in the study starting in either the 201 1-2012 or 
2012-2013 school year. 7 By design, Diplomas Now works in high-needs schools. The schools 
in the study come from high-poverty urban areas where students struggle academically and drop 


7 Five of the school districts are among the 20 largest in the country, and all but 1 are among the 100 larg- 
est. Chris Plotts and Jennifer Sable, Characteristics of the 100 Largest Public Elementary and Secondary 
School Districts in the United States: 2007-08, NCES 2010-349 (Washington, DC: U.S. Department of Educa- 
tion, National Center for Education Statistics, 2010). 
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out at high rates. The participating schools, all eligible for Title I funds, 8 serve large populations 
of low-income and minority students (80 percent eligible for free or reduced-price lunches; 83 
percent black and Hispanic). Furthermore, the high schools participating in the study have weak 
promoting power (56 percent), suggesting that they struggle to move students from ninth 
through twelfth grade on time. 9 Thirty-two of the participating secondary schools were random- 
ly assigned to implement the Diplomas Now model (DN schools) and 30 were assigned to con- 
tinue with “business as usual” (non-DN schools), either maintaining their existing practices and 
structures or pursuing other types of school reform. This random assignment design, often re- 
ferred to as the “gold standard” in evaluation, means that any differences between the DN and 
non-DN schools that emerge after random assignment can be attributed to the program; in short, 
Diplomas Now caused the observed differences. 

The study’s experimental design makes it possible for the evaluation to assess the early 
impact of Diplomas Now on students’ ABC outcomes during the first two years of the model’s 
implementation. For students who enter Diplomas Now in sixth or ninth grade, what difference 
does Diplomas Now make for three primary student outcomes: attendance rates (proportion of 
enrolled days in attendance), suspensions (in-school or out-of-school) and expulsions, and suc- 
cessful course completion? Essentially, does the implementation of Diplomas Now keep more 
students on track to high school graduation by the end of their middle school or high school 
transition years? The evaluation will also provide information about the longer-term impact of 
Diplomas Now: after four years, what is the impact of Diplomas Now on high school gradua- 
tion rates and on the ninth-grade success of students from Diplomas Now middle schools? 10 

The first two reports from the evaluation focus on the implementation of the Diplomas 
Now model, documenting how this complex, multicomponent reform intended to transform 
secondary schools is implemented by multiple partners. The implementation research explores 
what it takes to implement the model, what factors facilitate or hinder implementation, and the 
nature of the collaboration among multiple actors from the Diplomas Now organizations and 
the schools. The first report from the evaluation, released in 2014, focused on program start-up 
and first-year implementation in the DN schools. This second report continues the implementa- 
tion story, including findings about fidelity in the second year of model implementation. It also 
examines school structure and staff practice at DN and non-DN schools to see whether Diplo- 


8 Title I funds from the U.S. Department of Education go to schools with high numbers or high percentages 
of students from low-income families. 

9 William Corrin, Susan Sepanik, Aracelis Gray, Felix Fernandez, Ashley Briggs, and Kathleen K. Wang, 
Laying Tracks to Graduation: The First Year of Implementing Diplomas Now (New York: MDRC, 2014). 
“Promoting power” is calculated as the ratio of twelfth-graders to ninth-graders three years earlier. 

l0 While this report was being written, Johns Flopkins University was granted funding from the Office of 
Innovation and Improvement of the U.S. Department of Education to support an extension of the evaluation, 
which will make it possible to analyze the impact of Diplomas Now on student outcomes after four years. 
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mas Now is creating differences (that is, a “service contrast”) between the two groups of 
schools. Finally, it presents analyses of qualitative data that explore the context in which im- 
plementation was occurring, the integration of the model at schools, the importance of staff sta- 
bility, and program sustainability. 


Implementation of Diplomas Now in Year 2 

This report assesses the implementation of the Diplomas Now model during the second year 
and compares the findings from the second year with those from the first. It first looks at the 
fidelity with which the Diplomas Now model was implemented in the second year by the 32 
schools randomly assigned to do so. That is, did the model as implemented match the design? 
How much of the Diplomas Now school reform effort was put into place in these middle and 
high schools? Furthermore, it explores service contrast , or the extent to which the Diplomas 
Now intervention created differences between the practices of DN and non-DN schools. 

Fidelity of Implementation 

To create a measure of fidelity of implementation each of the Diplomas Now partner 
organizations detailed the components it felt were needed for full implementation of the model. 
Since this model as a whole has not yet been fully tested, it is not known which components are 
essential. Therefore, the fidelity of implementation score represents the model developers’ vi- 
sion of ideal implementation. Given the complexity of the model, 1 1 1 components were identi- 
fied. Each component is measured on a 0-1 scale with 0 equaling no or low implementation and 
1 equaling high fidelity to the model. The overall average fidelity score during the second year 
for all of the components across all of the DN schools is 0.62, suggesting that on average 
schools were implementing a majority of the components with high fidelity to the model, but 
that they still had room for growth to reach ideal implementation. This second-year implemen- 
tation score is similar to the first-year score discussed in a prior evaluation report, suggesting 
stability in program implementation even as some program components expanded into new 
grade levels during the second year. 

Several primary findings regarding fidelity emerged from the implementation analysis. 

• DN schools were most successful in obtaining, retaining, and training the 
auxiliary staff members needed to implement the model effectively (Pillar 
IV); using data to identify at-risk students; and collaborating to plan and pro- 
vide individual interventions for those students (Pillar III). 

Almost all schools had all of the necessary staff members in place to implement the 
model during Year 2 and program staff members from each of the partner organizations report- 
ed they had good access to each other. Implementation fidelity grew from Year 1 to Year 2 in 
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staffing and providing training to the auxiliary Diplomas Now staff, suggesting that Diplomas 
Now was better able in Year 2 to implement the components it had the most control over. Also, 
some staff members stayed from the first to the second year, making it easier to hire and support 
new staff members and ensuring that staff members were trained earlier and better. 

A large majority of schools were also successful during both years in establishing a data 
system to track students’ attendance, behavior, and course performance and ensure collabora- 
tion among staff members to plan and provide interventions for students falling off track. City 
Year AmeriCorps members were successful in providing extra and complementary services to 
students across all schools, but some schools were less successful in ensuring enough City Year 
AmeriCorps members were available to meet the needs of all the students. 

Schools were also moderately successful in establishing small learning communities of 
students that share the same classes and teachers (Pillar I), but many schools struggled to hold 
frequent meetings of these communities’ interdisciplinary teams of teachers. 

• DN schools were least successful at offering the peer coaching to teachers 
needed to strengthen practice, implementing curricular additions to ensure 
college and career readiness for all students (Pillar II), and involving parents 
and community members in school activities and decisions (Pillar IV). 

In both the first and second years of implementation, it was hard for schools to meet 
implementation goals in some areas where it was necessary to change school policy or structure 
or to get the school’s staff on board. Most schools struggled to achieve consistent coaching for 
math and English/language arts teachers at the level desired by Diplomas Now, and most 
schools also did not provide the prescribed academic foundations and accelerated remediation 
courses for struggling students. Finally, on average schools were not meeting the ideal levels of 
implementation for involving parents and community members. 

Service Contrast 

DN schools were more likely to implement the types of activities found under several 
of the Diplomas Now pillars than non-DN schools, suggesting that Diplomas Now is making 
DN schools different from non-DN schools. Unlike the fidelity of implementation score, there 
was quite a bit of growth from Year 1 to Year 2 in service contrast. Key findings include: 

• There was service contrast between DN and non-DN schools for the pillars 
that also showed strong and moderate implementation, including Pillar I 
(Teacher Teams and Small Learning Communities), Pillar III (Tiered Student 
Supports), and the part of Pillar IV (Can-Do Culture and Climate) focused on 
hiring and retaining the staff needed to implement the model. 
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Teachers at DN schools were more likely than teachers in non-DN schools to report 
teaching in extended class periods and collaborating with interdisciplinary teams that shared the 
same students; using data to identify at-risk students and meeting with other school staff mem- 
bers to plan interventions for them; and seeing students receive academic, behavioral, or emo- 
tional support. 

• Even though the implementation of teacher professional development and 
coaching did not fully meet the model’s goals, the service-contrast findings 
suggest that teachers at DN schools received more coaching than teachers at 
non-DN schools. However, teachers at DN schools reported similar levels as 
teachers at non-DN schools in the use of college readiness curricula. 

Math and English/language arts teachers at DN schools reported receiving more coach- 
ing than teachers at non-DN schools. Both groups of teachers reported similar, relatively high 
levels of adoption of academic reform curricula, incorporation of transitional support classes for 
struggling students, and use of student-centered and college and career readiness-focused strate- 
gies in the classroom, suggesting that many schools may have been implementing curricular 
refonns similar to those included in the Diplomas Now model. 

• The contrast between DN and non-DN schools increased from Year 1 to 
Year 2 in several areas, including collaboration of teachers within interdisci- 
plinary teams (Pillar I), professional development of teachers (Pillar II), and 
the use of data to identify struggling students (Pillar III). 

From Year 1 to Year 2 practices and structures aligned with the Diplomas Now model 
were maintained or slightly improved in DN schools, while similar structures and practices 
were less evident in non-DN schools in Year 1 and became even less evident in Year 2, suggest- 
ing that Diplomas Now helped stabilize resources and programs at the DN schools. 


Continued Challenges and Emerging Successes in Year 2 

Qualitative data were collected in nine DN schools to examine how schools’ contexts are affect- 
ing implementation, to explore the successes and struggles of implementing this complex pro- 
gram, and to investigate the sustainability of the Diplomas Now model. 

Factors External to the Diplomas Now Program 

Various factors external to the program (such as school closures, principal turnover, and 
budget cuts) affected the implementation of Diplomas Now. Since DN and non-DN schools 
were both likely to be affected by these issues, at least in some cases Diplomas Now was able to 
offer schools some consistency and additional support to deal with them. 
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Embedding Diplomas Now in Schools 

It emerged from the qualitative data that program staff members can foster stronger Di- 
plomas Now implementation at a school in two main ways: (1) aligning program goals with 
school priorities and (2) securing administrator and teacher support for the Diplomas Now mod- 
el. Case study data from Year 2 implementation indicated that Diplomas Now staff members 
better understood how to do those two things, and as a result were able to make Diplomas Now 
more “part of the school.” 

Challenges Caused by Diplomas Now Staff Turnover 

Despite gains from Year 1 to Year 2 in hiring and training program staff members, in 
some schools program staff turnover did cause setbacks in implementation and increased the 
burdens on the remaining staff members. Maintaining at least some consistency in Diplomas 
Now leadership preserved useful institutional knowledge about the successes and challenges of 
the first year of Diplomas Now implementation, and the collaborative relationships among part- 
ner organizations allowed for some flexibility to support consistent implementation. 

Sustainability of the Diplomas Now Model at the Schools 

The Diplomas Now plan for sustainability includes promoting local investment in City 
Year and Communities In Schools to maintain direct services to students in the long term while 
building schools’ ability to take charge of the school-wide refomis. Program sustainability was 
not yet a primary focus for school or program staff members, but some schools have made pro- 
gress toward sustaining elements of the model. Sustainability efforts include: maintaining data 
systems and meetings that identify struggling students and interventions to support them, train- 
ing school staff members to analyze data and lead these meetings, and promoting elements such 
as small learning communities and shared planning time to the school as a whole. Such efforts 
make features of the Diplomas Now model part of regular practice and school culture, and over 
time reduce the need for full support from Diplomas Now teams. 


Next Steps 

The implementation story told in this report sets the stage for the next report, which will provide 
evidence about the impacts of the Diplomas Now model. It will explore the impacts on school 
culture and climate, on student attitudes and behaviors, and on the key student outcomes of at- 
tendance, behavior, and course perfonnance that are the precursors to high school graduation 
and college and career readiness. 
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Chapter 1 

Introduction 


Background 

Although the national high school graduation rate has increased over the past decade, one in 
five students still do not complete high school in four years, and almost 30 percent of low in- 
come students fail to graduate on time. 1 Compared with high school graduates, dropouts are 
more likely to live in poverty, earn less money, suffer from poor health, be incarcerated, or be 
dependent on social services. 2 Students who face the most serious barriers to earning their di- 
plomas are in the greatest need of intensive academic, social, and other interventions to make it 
through high school, and most dropouts are concentrated in low-income, urban high schools. 

Research has shown that it is possible to predict a student’s likelihood of dropping out 
of high school using indicators of poor attendance, poor behavior, and course failure measured 
as early as middle school. 3 These findings suggest that programs may have greater success get- 
ting more students to graduation if they intervene with students who are off track as early as 
middle school. Moreover, ninth grade is a critical year, and researchers have shown that im- 
proving student course performance in the ninth grade can lead to substantial improvements in 
graduation rates. 4 

Among those students who do graduate high school, many do not graduate ready for 
college and need to take remedial (developmental education) courses: over 30 percent of col- 
lege undergraduates and over 40 percent of community college students enroll in such cours- 
es. 5 Beyond supporting struggling students to graduation, school improvement efforts need to 
ensure that all students participate and succeed in rigorous curricula that prepare them for col- 
lege and careers. 


The Diplomas Now Partnership 

Three national organizations — Talent Development Secondary, City Year, and Communities 
In Schools — have partnered to take on this combined task of school improvement and dropout 
prevention, and have created Diplomas Now. The Diplomas Now whole-school reform model 


'Mumane (2013); Balfanz et al. (2014). 

2 Moore (2014). 

3 Balfanz, Herzog, and Maclver (2007). 

4 Roderick, Kelley-Kemple, Johnson, and Beechum (2014). 

5 U.S. Department of Education, National Center for Education Statistics (2014). 
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seeks to transform secondary schools in high-poverty urban communities with large populations 
of academically low-perfonning students so that fewer students drop out and more students 
graduate high school prepared for college and careers. The Diplomas Now model is designed to 
support secondary education in some of the most underfunded communities in the country 
where poverty, lack of academic success, and high dropout rates are pervasive problems. The 
Diplomas Now model is a comprehensive approach that attempts to alleviate these stresses on 
schools through structural reform, instructional materials and curricula, teacher and administra- 
tor coaching and support, and an early warning system that identifies and targets students falling 
off the graduation track. The model brings additional people into the school to both support 
model implementation and directly provide additional assistance for students. 

The Diplomas Now partnership works with these schools to ensure that students are 
getting the support they need to (1) get to school and to class, (2) behave in ways that facilitate 
learning, and (3) keep up with the lessons being taught. In other words, the pathway to student 
success in Diplomas Now schools (DN schools) is linked to attendance, behavior, and course 
perfonnance, the “ABCs” that predict whether students graduate or drop out. The Diplomas 
Now partners collaborate to help schools provide the right services to the right students on time 
and at the right level of intensity by offering varying levels of support for students with different 
needs: whole-school support for all students, additional services for students showing early 
signs of falling off track, and student case management for students in need of deeper interven- 
tions. To determine which students need extra support, the approach relies on regular monitor- 
ing of Early Warning Indicator data, which include measures of students’ attendance, behavior, 
and course perfonnance. 

Talent Development Secondary focuses on whole-school organizational, instructional, 
and structural support and on facilitating the school’s use of an Early Warning Indicator (and 
intervention tracking) data system in regular interdisciplinary teacher team meetings. A school- 
based school transformation facilitator and English/language arts and math instructional coaches 
help school administrators and teachers implement model features related to school organiza- 
tion, classroom instruction, and the responsive use of data. City Year offers additional student 
support, with a program manager overseeing a team of City Year AmeriCorps members whose 
responsibilities include tutoring, mentoring, and classroom aid. Student case management is the 
responsibility of a Communities In Schools site coordinator, who manages a caseload of higher- 
risk, off-track students. 6 (See Box 1.1 for a more detailed description of the Diplomas Now 
partner organizations and their respective roles.) 


6 To the extent that the activities of each partner organization in Diplomas Now are similar to each of their 
independent models as implemented in other schools, the findings in this report specific to each organization 
may also inform their individual work. 


2 



Box 1.1 


The Diplomas Now Partners and Their Roles 
Talent Development Secondary 

Talent Development Secondary, based at Johns Hopkins University, provides organizational, 
instructional, curricular, and data support to schools intended to help all students achieve at 
high levels and prevent them from falling off track. This school-wide effort includes reorgan- 
izing students and teachers into small learning communities, providing professional develop- 
ment and coaching to strengthen teacher pedagogy, and supplying college and career prepara- 
tory course content. Talent Development Secondary employs a school transformation facili- 
tator who works with school leaders to develop a systematic school transformation plan, cre- 
ates and manages an Early Warning Indicator data tool, organizes frequent multidisciplinary 
teacher-team data-response meetings that use the tool to guide a multitiered student support 
process, and works closely with local or regional instructional facilitators to oversee instruc- 
tional and curricular reforms. School-based instructional coaches in English/language arts 
and math support teachers’ delivery of course content. For many students, whole -school or- 
ganizational and instructional reforms, referred to as Tier 1 interventions, are enough to keep 
them on track. However, for some students, the Early Warning Indicator data indicate that ad- 
ditional and more intensive services are necessary. City Year and Communities In Schools 
play leading roles in providing these additional services. 

City Year 

City Year is an AmeriCorps program through which young adults ages 18 to 24 participate in 
a year of national service. A team of 10 or more City Year AmeriCorps members is as- 
signed to a school, increasing the number of adults in a building paying attention to students 
and working with them both in- and outside of classrooms. The team is led by a City Y ear 
program manager and team leaders (typically second-year AmeriCorps members), and its 
members are trained to provide a variety of academic and behavioral interventions — referred 
to as Tier II support — intended to help students stay on track to graduate. These “near peers” 
(given their proximity in age to the students) serve as tutors, mentors, and role models, per- 
sonalizing the school experience of the students. In addition, the AmeriCoips members pro- 
vide after-school programs and help teachers by working with students during class time. 

Communities In Schools 

Through a school-based site coordinator, Communities In Schools, a national dropout- 
prevention organization, draws on school and community resources to organize services — 
referred to as Tier 111 support — intended to move the students at the highest risk of dropping 
out back on track to graduation. The site coordinator assesses the needs of a student, develops 
an individual case plan to address those needs, and connects the student to services aligned 
with the case plan. Examples include professional counseling on anger management for a stu- 
dent with behavioral issues or long-term tutoring with a subject-area expert for a student far 
behind in class. A site coordinator will also provide direct services — for example, leading 
student discussion groups on topics like conflict resolution or the transition to adulthood. 
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The Diplomas Now Model 

The Diplomas Now model is a multidimensional system of organizational and instructional re- 
forms and targeted student support services. The elements of the model are classified as nine 
“inputs,” some of which represent substantial interventions on their own, such as implementing 
a rigorous curriculum or setting up a tiered intervention system to identify at-risk students and 
tailor interventions to their specific needs. Diplomas Now integrates these component interven- 
tions into a cohesive model focused on ensuring that all students have a path to graduation. 
Eight of these inputs are implemented in collaboration with school staff members and align with 
the Four Pillars of Diplomas Now, a characterization of the model used by Diplomas Now staff 
members to help them organize their work. The Four Pillars and their associated inputs are pre- 
sented in Figure 1.1. The ninth input consists of professional development for the staff of the 
Diplomas Now organizations, intended to give these Diplomas Now staff members the 
knowledge and skills they need to implement the Four Pillars. 

Pillar I: Teacher Teams and Small Learning Communities 

This pillar is focused on school structure and organization. Diplomas Now collaborates 
with school leaders to organize schools such that small groups of teachers work with the same 
population of students. These small learning communities create opportunities for personaliza- 
tion where teams of teachers know the same students and can work together to best teach and 
support them. Students also share the same classes, and become known to one another. These 
teacher teams and small learning communities function best when there are opportunities for 
teachers to collaborate within the daily schedule and when they have classes long enough to 
cover material in depth and keep up the pace of instruction. 

Pillar II: Curriculum and Instruction with Professional Development 

This pillar is focused on teaching and learning, and on giving teachers the training and 
resources they need to deliver strong lessons. Through professional development activities and 
instructional coaching, teachers have an opportunity to sharpen their pedagogy. Having curricu- 
lar materials aligned with college- and career-ready standards means that these teachers have 
useful content to deliver to students. This pillar also includes professional development and cur- 
ricular materials for accelerated remediation courses for struggling students. 

Pillar III: Tiered Student Supports 

Providing more intensive support for students with greater needs is the core idea of this 
pillar. The tiered intervention model involves implementing an early warning system that draws 
on data on the ABC indicators for individual students. It relies on staff members who work with 
a group of students in common having regular times to meet to review those data and to plan 
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interventions for students who are off track or at risk of going off track. In the early years of 
implementation, the Talent Development school transformation facilitator helps organize the 
Early Warning Indicator data, introduces the idea of a tiered system of responses to student 
needs, and facilitates the meetings. Once necessary interventions have been identified for a stu- 
dent, City Year and Communities In Schools staff members at the school help provide and 
manage that extra support. Diplomas Now staff members and their school staff partners also pay 
attention to trends in the data that might suggest that a school needs to make classroom, grade- 
level, or school-wide improvements to address issues related to student attendance, behavior, 
and course performance. 

Pillar IV: Can-Do Culture and Climate 

School reform is difficult, and school staff members often have too much to do when 
they are asked to effect change. This pillar is about providing a school’s staff with additional 
resources to make change feel possible. Diplomas Now brings over a dozen staff members to a 
school to help coordinate school transformation, introduce new practices and structures, provide 
training and support to school staff members, provide additional services to students, and en- 
gage families and community organizations. All of these staff members are trained by Diplomas 
Now before and throughout the school year. Providing and organizing resources to assist the 
school’s staff helps foster a culture and climate where it feels possible to improve the school and 
support students better. 

Program Staff Training and Development 

One model input, Program Staff Training and Development, involves providing the 
skills and knowledge to Diplomas Now staff members that they need to implement the other 
eight model inputs under the Four Pillars. This includes training in the summer and during the 
school year for school-level Talent Development Secondary, City Year, and Communities In 
Schools staff members. To ensure successful implementation at each school, each of the Di- 
plomas Now partner organizations has a system of support services for school-based staff 
members that includes locally based and national program experts with strong relationships to 
school districts . 7 


Additional information about Diplomas Now can be found on the Internet at http://diplomasnow.org. The 
Diplomas Now website includes information about the partner organizations and the model, as well as contact 
information for the partnership. A more detailed discussion of the model can also be found in the prior evalua- 
tion report, along with discussion of the national and local organizational structure of Diplomas Now (Corrin et 
al., 2014a). 
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Phases of Model Implementation 

Many of the Diplomas Now model inputs are phased in by grade level over a couple of 
years. During the first year of implementation, certain Diplomas Now services were rolled out 
in the sixth grade in middle schools and the ninth grade in high schools. For example, the ser- 
vices in Pillar III (Tiered Student Supports) to identify at-risk students and offer them special- 
ized assistance were concentrated in the sixth and ninth grades during the first year in most DN 
schools. In the second year of implementation, these services followed the students receiving 
them into the seventh and tenth grades, but were also offered to the incoming cohort of sixth- 
and ninth-grade students. 

The program is basically identical for sixth- and seventh-grade students in the middle 
schools. There can be some programmatic differences between the ninth and tenth grades in the 
high schools. Ninth grade is a particularly tough year for many students as they adjust to a new 
school and new expectations. At the same time, success during this transition year can lay an 
important foundation for students’ ability to graduate. For these reasons, the high school portion 
of the Diplomas Now model is at its most intensive during the ninth grade, and services are 
slowly reduced as the students move into upper grades. This plays out differently in different 
schools, but only ninth-graders participate in a student support curriculum (known as Freshman 
Seminar) and City Year AmeriCorps members may provide services to a smaller number of 
tenth-graders or offer support in fewer subjects during the tenth grade. 

In some cases, limitations in program funding may have led to differences between 
grades at both middle schools and high schools. During the start-up year, implementation costs 
totaled approximately $491,000 per school. In the second year, the average cost per school was 
also approximately $491,000, but as some start-up costs diminished the program was also ex- 
panding to include the seventh and tenth grades. 8 In schools where funding was limited during 
Year 2, the Diplomas Now partners focused resources on the transition years (sixth and ninth 
grades), and followed the seventh- and tenth-grade students to the extent possible using any re- 
sources still available. 


8 Talent Development Secondary’s costs were $246,400, which covered curricular materials, the school 
transformation facilitator placed at the school, and technical assistance from mathematics and English/language 
arts instructional facilitators and from a school and student support services facilitator (who worked across all 
implementing schools in a district). City Year’s costs after supplements from AmeriCorps and private funding 
were about $145,700 for the City Year team of near peers and an on-site program manager. Communities In 
Schools’ costs averaged $99,000 for the Communities In Schools site coordinator, service provision, and infra- 
structure support. School-level instructional coaches were often an in-kind contribution from the school dis- 
tricts. In the few cases where they were not, those costs were included in Talent Development Secondary’s 
costs (and are reflected in the Talent Development Secondary average cost figure above). 
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Hypothesized Outcomes 

Implementation of the nine inputs (in the Four Pillars) is hypothesized to affect a series 
of outcomes (see Figure 1.1). Initially, model implementation is supposed to lead to early-stage 
changes in school outcomes such as the quality of school climate and communications among 
stakeholders, and in student outcomes like study habits and attitudes about and engagement 
with school. These changes, in turn, are expected to lead to impacts on intemiediate outcomes 
— the ABC indicators of attendance, behavior, and course performance. Positive impacts on 
ABC outcomes should then lead to increased high school graduation rates. 


The National i3 Evaluation of Diplomas Now 

Johns Hopkins University, home to Talent Development, was awarded a federal Investing in 
Innovation (i3) validation grant in 2010 to support the expansion and evaluation of Diplomas 
Now. The five-year i3 validation grant supports the expansion of Diplomas Now from a few 
schools to more than 30 schools across more than 10 school districts. The grant funds also sup- 
port a rigorous experimental evaluation of the Diplomas Now model, the results of which, if 
positive, will “validate” Diplomas Now as an effective secondary school reform model. The 
evaluation, being conducted by two evaluation research firms, MDRC and ICF International, 
explores not only the impact of Diplomas Now but also its implementation, providing lessons to 
the field about what it takes to implement the model and how it rolls out in different school and 
district contexts. 

The evaluation has a school-level random assignment design. Schools were randomly 
assigned to implement Diplomas Now (DN schools) or to continue with business as usual (non- 
DN schools), either continuing existing practices and structures within their schools or pursuing 
other types of school reform. This random assignment design, often referred to as the “gold 
standard” in evaluation design, creates circumstances under which any differences between the 
two groups of schools (DN and non-DN) that emerge after random assignment can be attributed 
to the program (Diplomas Now): Diplomas Now caused the observed differences. 

As noted earlier, the Diplomas Now model is designed to bring resources and support to 
urban communities struggling with high levels of poverty, low student academic performance, 
and a prevalence of high school dropout. Sixty-two schools (33 middle schools and 29 high 
schools) from 1 1 large urban school districts across the country were recruited to participate in 
the study starting in either the 2011-2012 or 2012-2013 school year. Five of the districts are 
among the 20 largest school districts in the country and 10 are among the 100 largest, measured 



by numbers of students served. 9 The participating schools, all eligible for Title I funds, 10 serve 
large populations of low-income and minority students (80 percent eligible for free or reduced- 
price lunches; 83 percent black and Hispanic). Furthermore, the high schools participating in the 
study also have weak promoting power (56 percent), suggesting that they struggle to move stu- 
dents from ninth through twelfth grade on time. 11 Essentially, the schools in the study sample 
represent some of the most challenged urban secondary schools in the country, all operating in 
large, underfunded school districts. 

Random assigmnent resulted in 32 DN schools and 30 non-DN schools. 12 Many of the 
measurable characteristics of the two groups of schools were compared: their sizes, staff sizes, 
and student populations, as well as the types of programs they had available before the start of 
the evaluation. These comparisons indicated that the two groups of schools were similar to each 
other, suggesting that the random assigmnent of schools was successful and the non-DN schools 
provide a convincing representation of what would have happened in the DN schools had they 
not implemented the intervention. 13 

One focus of the evaluation is the implementation of the Diplomas Now model. The 
implementation research describes this complex refonn, explores what it takes for multiple 
partners to implement the model, and assesses the fidelity with which the model is implemented 
(that is, how close the program elements as adopted by schools came to the designers’ intent). In 
addition, this research looks at factors that facilitate or hamper implementation, the nature of the 
collaboration among multiple actors from the Diplomas Now organizations and the implement- 
ing schools, and the degree to which early implementation sets up the model for longer-term 
sustainability. This line of research also investigates whether DN schools change in ways that 
make them different from non-DN schools. 

As discussed earlier, the Diplomas Now model seeks to address the needs of students 
who exhibit indicators of falling off track (poor attendance, negative school behaviors, and 
course failure), particularly during transition years when students are most vulnerable to get- 
ting off track: sixth grade in middle schools and ninth grade in high schools. In the longer term, 
the goal of the Diplomas Now model is to prepare middle school students to have a successful 


9 Plotts and Sable (2010) report various statistics about the 100 largest school districts in the United States. 

"'Title 1 funds from the U.S. Department of Education go to schools with high numbers or high percent- 
ages of students from low-income families. 

'(orrin et al. (2014a). “Promoting power” is calculated as the ratio of twelfth-grade students to ninth- 
grade students three years earlier (for example, the ratio of twelfth-graders in 2010-2011 to ninth-graders in 
2007-2008). 

12 One school closed after the first year of implementation and all data from that school are missing in Year 2. 

l3 The first report from this evaluation includes detailed information about the comparison of “baseline” 
characteristics of the DN and non-DN schools (Corrin et al., 2014a). 
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transition into high school and to increase the number of high school students who graduate. 
Therefore, the evaluation also investigates the impact of Diplomas Now on student outcomes. 
For students who enter Diplomas Now schools in sixth and ninth grade, what is the impact of 
Diplomas Now on three primary student outcomes: attendance rates (proportion of enrolled 
days in attendance), suspensions (in-school or out-of-school) and expulsions, and successful 
course completion? Essentially, does the implementation of Diplomas Now keep more stu- 
dents on track to high school graduation by the end of their middle school or high school tran- 
sition years? 

While this report was being written, Johns Hopkins University was granted funding 
from the Office of Innovation and Improvement of the U.S. Department of Education to support 
an extension of the evaluation. The MDRC and ICF research team will now be able to follow 
students for four years, thus fracking ninth-grade students through four-year high school gradua- 
tion and sixth grade students through their next transition year — ninth grade in high school. 
Thus, the evaluation will also provide information about the longer-term impact of Diplomas 
Now on the graduation of students in Diplomas Now high schools and on the ninth-grade suc- 
cess of students from Diplomas Now middle schools. 


Summary of Key Findings from the First Year of Implementation 

The first report from the evaluation, released in 2014, focused on program start-up and first-year 
implementation in the DN schools. The report introduced its readers to the Diplomas Now 
model, described the schools and school districts participating in the evaluation, shared findings 
about first-year implementation fidelity in the DN schools, and discussed how program and 
school staff members collaborated to implement the DN model. The prior report shared the fol- 
lowing key findings and lessons. 

• During the first year of implementation, on average model implementation in 
the 32 DN schools had gotten under way and gained some traction. During 
the first year of implementation, DN schools had a fidelity of implementation 
score of 0.61 on a scale of 0 to 1, where 1 would represent a high level of fi- 
delity to the model on all 111 program components across the nine inputs. 

Complex, multifaceted whole-school refomis like Diplomas Now typically take a few 
years to reach full implementation. Getting the majority of model components implemented in 
the first year suggested that schools were off to a promising start with this refonn. 

• However, none of the DN schools were able to implement all of the 62 com- 
ponents of the model believed to be most critical by the Diplomas Now or- 
ganizations in the first year. 
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The Diplomas Now Implementation Support Team identified a subset of the program 
components that it hypothesized were the most likely to lead to the desired changes in school 
and student outcomes. 14 During the first year of implementation, the DN schools struggled to 
implement some of these “critical” components. For example, some schools did not have all 
their Diplomas Now program staff members in place by the start of the school year; only about 
half of the schools managed to hold Early Warning Indicator meetings weekly; and although 
instructional support for English/language arts and math teachers was provided at many schools, 
instructional coaches struggled to work with teachers as often as the model design expected. 

• DN schools were most successful during the first year in implementing Pillar 
III, particularly in incorporating the tiered intervention model and additional 
student support services — like tutoring and after-school programs — into 
the existing school structure. 

Across DN schools, the Tiered Intervention Model, Integrated On-Site Support, and 
Student Supports were among the most fully implemented program inputs during the first year 
of implementation. Some schools had activities associated with these inputs in place prior to 
Diplomas Now program implementation, such as a tiered system of support services and com- 
mon planning time. 

• DN schools were less successful in implementing components of Pillar II, 
particularly new curricula and peer coaching models, which both require 
gaining the trust and investment of school administrators and teachers, and 
may require additional time to implement as a result. 

Qualitative data revealed that school staff members were not always convinced of the 
value of implementing new curricula, and some teachers were reluctant to be “coached.” These 
findings speak to the importance of giving a school’s staff a role in selecting some program staff 
members, if possible, to encourage greater rapport and trust between school and Diplomas Now 
staff members. This is especially the case for instructional coaches, who frequently interact with 
teachers on a one-on-one, peer-to-peer basis. 

The evaluation team also used interview and focus group data to explore collaborations 
during the first year of implementation, since effective collaboration is at the heart of the com- 
plex Diplomas Now school reform. 

• Two aspects of collaboration appeared to be most important across schools 
and various role groups: role clarity and school staff investment. 


14 The Diplomas Now Implementation Support Team is a group of national representatives from the three 
partner organizations who provide guidance about model implementation to local Diplomas Now teams. 
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Administrators and teachers are key stakeholders whose engagement in implementing 
the Diplomas Now model inputs is essential, and in order for them to become actively engaged 
in the model they must first understand it. They may leam about it through information ses- 
sions, by meeting school staff members at other DN schools, or in other ways. Continual com- 
munication, including regular meetings and informal check-ins, helps build the trust and ac- 
ceptance necessary for the collaborative work of model implementation. 

In addition, it is crucial that Diplomas Now school-based staff members establish their 
purpose and roles, with teachers and administrators as well as among themselves. The influx of 
Talent Development, City Year, and Communities In Schools staff members provided schools 
with the human resources necessary for the implementation of the Diplomas Now model. How- 
ever, increased clarity regarding the roles and responsibilities of these Diplomas Now staff 
members might have Ilirther improved model implementation in the first year. 


Orientation to This Report 

This report continues the implementation story begun in the first report. It includes findings 
about implementation fidelity in the second year of model implementation. It also looks at dif- 
ferent characteristics of school structure and practice at both DN and non-DN schools to see 
whether the implementation of the Diplomas Now model is creating differences (that is, a “ser- 
vice contrast”) in how the two groups of schools are organized and how their respective staffs 
function. 

The next chapter presents information about model implementation in Year 2, discuss- 
ing both the fidelity with which the model was implemented and service contrast. The third 
chapter then discusses changes in implementation over time, comparing Year 1 and Year 2 find- 
ings about implementation fidelity and service contrast. These two chapters focus on the pro- 
gram inputs on the left-hand side of Figure 1.1 and discuss model implementation by pillar. Da- 
ta regarding the fidelity with which the DN schools implemented the model come from Diplo- 
mas Now staff surveys, which were developed to measure specific components in each input. 
The staff members surveyed include: the Diplomas Now Implementation Support Team, Talent 
Development school transformation facilitators, City Year program managers, and Communi- 
ties In Schools site coordinators. Data regarding the contrast in services between the DN and 
non-DN schools come from surveys of teachers and administrators (principals and assistant 
principals) at both DN and non-DN schools. All surveys were administered in the spring of each 
year. 


The fourth chapter presents analyses that explore the context in which implementation 
was occurring, the integration of the model at schools, the importance of staff stability, and pro- 
gram sustainability. This chapter draws on qualitative data collected through interviews and fo- 
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cus groups at nine case study schools from four school districts. The chapter highlights common 
themes across participants and schools and provides illustrative examples from individuals or 
selected schools. The final, concluding chapter ties the various analyses together and sets up the 
next report. 


Future Evaluation Reports 

Five reports are expected from this evaluation. The next report will present the findings from 
analyses of the impact of Diplomas Now on (1) early school and student outcomes, such as cre- 
ating a positive school climate and strengthening student confidence and persistence in school 
(see “early” outcomes in Figure 1.1), and (2) ABC outcomes in the second year of implementa- 
tion (see “intennediate” outcomes in Figure 1.1). The fourth report will investigate variation in 
implementation and impacts, as well as relationships between implementation and outcomes. 
The fifth and final report will present the results of analyses of the impact of Diplomas Now on 
the longer-tenn outcomes of high school graduation rates for students who entered the model in 
ninth grade, and ninth-grade success for students who entered in sixth grade. 
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Chapter 2 


Implementation of the Diplomas Now Model 

During Year 2 


This chapter focuses on the implementation of the Diplomas Now model during the second year 
of the program, considering it in two different ways. First the chapter considers fidelity of im- 
plementation: how actual implementation matched the Diplomas Now model and varied across 
schools. Since this is the first time the Diplomas Now model is being rigorously tested, it is not 
yet known what level of model implementation leads to positive change at a school, and ulti- 
mately to positive effects on student outcomes. Second, to illuminate how Diplomas Now is 
changing school practices and structures, the chapter examines service contrast: the extent to 
which Diplomas Now is creating differences between the schools that were assigned to imple- 
ment the model (DN schools) and the schools that were not (non-DN schools). Because this 
assignment was random, the non-DN schools offer insight into how the DN schools might have 
looked if they had not implemented Diplomas Now. 

This chapter offers several main findings: 

• The overall fidelity score of 0.62 on a scale of 0 to 1 suggests that on average 
schools were implementing a majority of the components with fidelity to the 
model, but still had room for growth to reach ideal implementation. 

• During Year 2, fidelity of implementation findings suggest that DN schools 
were most successful in setting up the Diplomas Now support staff in the 
schools, ensuring that data were used to identify student needs and plan for 
interventions, offering specialized support both inside and outside of class- 
rooms to students who needed it, creating small learning environments where 
interdisciplinary teams of teachers worked with small groups of students, and 
establishing extended class time for core courses. 

• Across all these areas, the programs offered in DN schools were more inten- 
sive than those found in non-DN schools. That is, implementation of the 
model was creating a contrast between the structures and practices of the two 
groups of schools. 

• On average DN schools were less likely to meet the goals the model set for 
the use of reform curricula, the availability of additional support courses for 
struggling students in math and English/language arts classes, the level of in- 
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structional coaching offered to math and English/language arts teachers, and 
the engagement of parents and community members. 

• There was also little contrast between DN and non-DN schools in their use 
of college readiness curricula and their family and community involvement, 
although teachers in DN and non-DN schools alike reported relatively high 
use of college readiness curricula and accelerated remedial courses for 
struggling students. Even though DN schools had trouble meeting the im- 
plementation goals of the model for instructional coaching of teachers, 
teachers at DN schools still reported receiving more peer coaching than their 
non-DN counterparts. 


Data Sources and Analyses 

To capture yearly fidelity to the model’s implementation goals, surveys were administered to 
Diplomas Now program staff members connected with each school. The four surveys adminis- 
tered in the spring of each implementation year were: (1) the Diplomas Now Implementation 
Support Team survey, (2) the Talent Development school transfomiation facilitator survey, (3) 
the City Year program manager survey, and (4) the Communities In Schools site coordinator 
survey. 1 Implementation fidelity measures were developed in collaboration with the Implemen- 
tation Support Team to capture the various components of each input within each pillar listed in 
Figure 1.1. Across the pillars and inputs 1 1 1 components were identified, and a scale from 0 (no 
or low implementation) to 1 (high implementation) was established for each component. The 
scores for each component were averaged within each input, and the average of the nine input 
scores then gave the overall score for a DN school, a proportion of how much or how “fully” 
schools implemented the Diplomas Now model. (See Appendix A for a detailed description of 
how the fidelity scores were calculated.) Input-level scores were also averaged across all DN 
schools to obtain the average fidelity scores for each input found in Table 2.1. 

The Implementation Support Team also established a fidelity score for each compo- 
nent that it considered “adequate” implementation (varying among components from 0.25 to 1 
on the scale), and the research team calculated the percentage of schools to achieve adequate 
fidelity to each component. This chapter reports the number of components in each input that 


See Appendix A for additional detail on the program staff surveys and response rates. The Diplomas 
Now Implementation Support Team is a group of national representatives from the three partner organizations 
who provide guidance about model implementation to local Diplomas Now teams. Diplomas Now staff mem- 
bers also collected program documents, databases, and service records to supplement program staff surveys. 
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Diplomas Now 
Table 2.1 


Fidelity of Implementation Findings, Year 2, 

All DN Schools, Schools in the Top Quartile, and Schools in the Bottom Quartile 


Model Inputs 

All DN 
Schools 

Average Fidelity Score 
Top Bottom 

Quartile Quartile Difference 

Pillar I. Teacher Teams and Small Lear nine Communities 

0.69 

0.85 

0.47 

0.38 *** 

Strong Learning Environments ’ 

0.69 

0.85 

0.47 

0.38 *** 

Pillar II. Curriculum and Instruction with Professional Development 

0.37 

0.67 

0.15 

0.52 *** 

Professional Development and Peer Coaching 3 ' 13 

0.42 

0.84 

0.05 

q 7 g *** 

Curriculum for College Readiness 13 

0.31 

0.50 

0.25 

0.25 ** 

Pillar III. Tiered Student Supports 

0.69 

0.76 

0.56 

0.21 *** 

Tiered Intervention Model 3 

0.75 

0.96 

0.45 

0.51 *** 

Student Supports 3 

0.73 

0.78 

0.67 

0.11 ** 

Student Case Management 3 

0.59 

0.56 

0.55 

0.01 

Pillar IV. Can-Do Culture and Climate 

0.69 

0.80 

0.59 

0.21 *** 

Integrated On-Site Support 3 

0.87 

0.90 

0.82 

0.07 

Family and Community Involvement 

0.51 

0.70 

0.35 

0.35 *** 

Program Staff Training and Development 

0.71 

0.80 

0.59 

0.20 *** 

Overall 

0.62 

0.76 

0.47 

0.30 *** 

Sample size 

31 

7 
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(continued) 



Table 2.1 (continued) 

SOURCE: Diplomas Now fidelity of implementation program staff surveys, 2012, 2013, and 2014. 

NOTES: Each of the nine inputs consists of a set of more specific components measured on a 0-1 scale. For each 
DN school, all of the component scores under an input are averaged to create the implementation score for that 
input. These scores are then averaged to create the pillar-level and overall implementation scores. 

Using the overall implementation score for each school, the top quartile and bottom quartile of implementing 
schools were identified. The "top quartile" column is the average implementation scores of the seven schools 
that make up the top quartile and the "bottom quartile" column is the average implementation scores of the eight 
schools that make up the bottom quartile. The "difference" column is the difference between the average 
implementation scores for the top and bottom quartiles. 

One DN school closed prior to the second year of implementation and is not included in this analysis. 

A two tailed t-test was applied to differences. Statistical significance levels are indicted as follows: *** =1 
percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

indicates model inputs designated as critical to the Diplomas Now model. 

b Strong Learning Environments, Professional Development and Peer Coaching, and Curriculum for College 
Readiness include some components specific to either middle schools or high schools. 

were adequately implemented by more than 80 percent of schools and the number of compo- 
nents in each input that were adequately implemented by fewer than 50 percent of schools. 
(Appendix Table A.3 shows the percentages of schools that adequately implemented each of 
the 1 1 1 components.) 

As noted earlier, it is not yet known what level of implementation is needed to actually 
effect change in schools. For this reason, it is also important to understand how what is happen- 
ing at DN schools differs from what would be happening in the absence of Diplomas Now. To 
measure the service contrast between DN and non-DN schools, teachers and administrators 
(principals and assistant principals) at all study schools were also surveyed each spring. 2 Re- 
sponses of teachers and administrators at DN schools are compared with those of teachers and 
administrators at non-DN schools. 3 These analyses use data only from middle school teachers 
who taught sixth or seventh grade and high school teachers who taught ninth or tenth grade, 
since these grades were the most fully served by Diplomas Now during the second year of im- 
plementation. 4 All measures in these analyses were created using one or more items from the 
surveys. 5 Although the service contrast measures are aligned with the Diplomas Now model, 

2 See Appendix B for additional detail on the teacher and administrator surveys and response rates. 

3 The analytic model used in these analyses is similar to the analysis method planned for the confirmatory 
outcomes of student attendance, behavior, and course performance to be presented in the next report. See Ap- 
pendix B for details on the analytic model. 

4 Appendix B includes analyses of teachers who taught sixth and ninth grade in the first and second year of 
implementation. Sixth and ninth grades were the focus of the first year of implementation, and these teachers 
were therefore more likely to have participated in two years of model implementation by the spring of the sec- 
ond year than teachers who did not teach these grades. 

5 Where more than one survey item was averaged to create a composite measure, factor analyses were used 
and Cronbach’s alphas are all above 0.7. 
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the survey items from which they were calculated were phrased generally enough so that re- 
spondents at non-DN schools could still interpret them in reference to their schools. 


Comparison of Fidelity of Implementation and Service Contrast 

It might be expected that fidelity of implementation and service contrast would correlate with 
one another, since both analyses are measuring aspects of the implementation of the Diplomas 
Now model, albeit in different ways. Fidelity is a measure of implementation that examines 
how fully DN schools met a set of goals set by the model developers. Service contrast is a rela- 
tive measure that evaluates how much DN schools differed from non-DN schools in the imple- 
mentation of activities aligned with the Diplomas Now model. Thus, when DN schools have 
high fidelity of implementation, it creates more “room” for a difference in practices or structures 
to emerge between DN and non-DN schools. Alternatively, when DN schools have low fidelity 
of implementation, there is less “room” for DN schools to differentiate themselves from non- 
DN schools. 

However, there are also circumstances under which fidelity of implementation and ser- 
vice contrast may not be correlated. The fidelity of implementation analysis does not take into 
account what types of structures and school practices are being implemented in the non-DN 
schools. High fidelity scores at DN schools may be the result of practices or reforms that are 
easier to implement and so more likely to also be well implemented in non-DN schools, thus 
leading to less contrast between the two groups of schools. Conversely, low fidelity scores may 
reflect changes that are more challenging to make. This could be true in both DN and non-DN 
schools. Therefore, while DN schools may not have been fully successful in implementing 
components, non-DN schools could have been even less successful or not have been imple- 
menting similar practices at all, which would create some service contrast. 

It is also worth noting that in this case the fidelity scores are based on surveys of the 
Diplomas Now staff members at the schools, while the service contrast measures are based on 
surveys of teachers and administrators, who have a different perspective. In addition, while fi- 
delity of implementation was scored for all 111 program components of this complex model, 
the measures of service contrast only include the types of programs that could be identified by 
teachers and administrators in both DN and non-DN schools. 
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Implementation of the Diplomas Now Model by Pillar 

The first column of Table 2.1 provides the average fidelity score by pillar and the average fideli- 
ty score by input across the entire sample of DN schools. 6 On average across all inputs and pil- 
lars, the second-year fidelity score (0.62) indicates that the majority of program components 
were implemented at a high level of fidelity to the model, a moderate level of implementation 
very similar to what occurred in the first year. 7 

Pillar-level Year 2 fidelity scores suggest that over two-thirds of the components were 
implemented at a high level of fidelity for three of the four pillars: Pillar I (Teacher Teams and 
Small Learning Communities, with a score of 0.69), Pillar III (Tiered Student Supports, 0.69), 
and Pillar IV (Can-Do Culture and Climate, 0.69). Among individual inputs, Integrated On- 
Site Support (0.87), Tiered Intervention Model (0.75), and Student Supports (0.73) were the 
most fully implemented. These are elements that represent the newest technology, using Early 
Warning Indicator data to identify struggling students, and came with the greatest amount of 
implementation support from Diplomas Now, including additional staff members brought into 
the schools. 

In contrast, Pillar II (Curriculum and Instruction with Professional Development) was 
the most difficult to implement (with a score of 0.37), particularly Curriculum for College 
Readiness, which received the lowest implementation score (0.31) of any input. This may be in 
part because some schools believed they already had in place curricula mandated or supported 
by the district that were functionally equivalent to those implemented as part of Diplomas Now. 
Teacher instructional coaching was also hard to implement because it required a high level of 
acceptance from teachers and school administrators and because resources to support instruc- 
tional coaches often came from the district, whereas funding for other Diplomas Now staff 
members generally came from outside sources. 

The ninth input, Program Staff Training and Development, is about providing Diplomas 
Now staff members with the skills and knowledge they need to implement the other eight in- 
puts, and does not fall within the pillar structure. The fidelity score for this input suggests suc- 
cessful implementation (0.71) on average. A majority of schools reported adequate levels of 
implementation of all but 1 of the 18 components in this input. It is worth noting that staff 
members from each of the partner groups reported high levels of access to and communication 
with the staff members from the other partner organizations at their schools, which should have 
helped the three partner organizations become integrated. 


6 See Appendix A for a kill description of how fidelity scores were calculated overall and by pillar and input. 

7 There were no significant differences in overall fidelity scores between middle and high schools or between 
schools that joined the study in the 2011-2012 school year and those that joined in the 2012-2013 school year. 


20 



Table 2.1 also notes which inputs were designated as critical to successful early imple- 
mentation by the Diplomas Now Implementation Support Team. Two of the three inputs with 
the lowest implementation scores (Curriculum for College Readiness and Family and Commu- 
nity Involvement) were also identified as lower implementation priorities. 

DN schools were ranked by their overall implementation scores. Then, for each pillar 
and input the schools with the greatest overall implementation success (those whose total aver- 
age scores were in the top 25 percent, or quartile) were compared with those with the least over- 
all implementation success (those in the bottom quartile). This comparison reveals which inputs 
were challenging to implement both for schools doing the best with the model and for those 
struggling the most. Those aspects of the model must be universally difficult to implement. 
Meanwhile, some inputs are commonly found implemented with high fidelity in schools in the 
top quartile overall but are less often found implemented with high fidelity in schools in the bot- 
tom quartile. Those aspects of the model may be areas where schools could benefit from addi- 
tional implementation support. 

The last three columns of Table 2.1 display the average implementation score for the 
top quartile of schools, the average score for the bottom quartile of schools, the difference be- 
tween those average scores, and whether that difference is statistically significant. Implementa- 
tion scores for schools in the top quartile averaged 0.76 (approximately three-fourths of model 
components implemented with high fidelity) and ranged from 0.74 to 0.80; scores for schools in 
the bottom quartile averaged 0.47 (just under half of model components implemented with high 
fidelity) and ranged from 0.36 to 0.53. Professional Development and Peer Coaching, Tiered 
Intervention Model, and Strong Learning Environments show the largest differences between 
schools in the top and bottom quartiles, suggesting that these aspects of the program were hard- 
er to implement in some schools than others. These are all areas where changes to school struc- 
ture or teacher practice are important to successful implementation. The smallest differences 
between schools in the top and bottom quartiles are found in Student Case Management and 
Integrated On-Site Support. For both of these inputs, successful implementation is less con- 
strained by school structure and teacher practice and more the result of the efforts of Diplomas 
Now staff members. 

The discussion of input-level implementation fidelity and service contrast that follows 
is organized according to the Four Pillars of the Diplomas Now model. Each section discusses 
input-level fidelity of implementation for one of the pillars. 

These findings are reported by pillar in Tables 2.2, 2.4, 2.6, and 2.8. Each section identi- 
fies the individual components within each input that were consistently implemented (that is, 
those implemented adequately by 80 percent or more of DN schools) and those that were incon- 
sistently implemented (that is, those implemented adequately by fewer than 50 percent of DN 
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schools). For each input, the tables list the total number of components along with the number 
of components that were consistently implemented (that is, those implemented adequately by 80 
percent or more of DN schools) and the number that were inconsistently implemented (that is, 
those implemented by fewer than 50 percent of DN schools). The tables also provide examples 
of components in each input that schools either implemented consistently or struggled with. 8 

The discussion of implementation fidelity for each input is followed by a discussion of 
pertinent service contrast findings. Tables 2.3, 2.5, 2.7, and 2.9 compare findings from surveys 
of teachers at DN and non-DN schools. These tables include the average score on each measure 
for the DN schools, the average score for the non-DN schools, and the difference between the 
scores. The effect size and p-value are also included. The effect size is calculated by dividing 
the raw difference between scores by the standard deviation of the measure. Since the survey 
measures use different units (some are on a 10-point scale, for example, while others are “times 
per month”), the effect size allows for comparisons among them. The p-value shows the proba- 
bility that this difference arose by chance (the smaller the p-value, the less likely the difference 
arose by chance). Measures marked with asterisks are considered statistically significant, those 
where one may conclude with some confidence that there was a contrast in services between 
DN and non-DN schools. The more asterisks, the more confident one can be. 

Pillar I: Teacher Teams and Small Learning Communities 

The input aligned with this pillar is “Strong Learning Environments.” This section dis- 
cusses the following key findings. 

• Schools implemented almost all the components of this input with a moder- 
ate level of success, and teachers at DN schools reported a higher frequency 
of participation in the types of activities found under this input than teachers 
at non-DN schools. 

• Although not universally implemented in all DN schools, extended or longer 
class periods of 70 to 90 minutes were more likely to occur in DN schools 
than in non-DN schools. 

• Although most DN schools struggled to meet the expectation of the Diplo- 
mas Now model that interdisciplinary teams of teachers would meet multiple 
times per week to discuss their shared group of students, teachers at DN 
schools did report meeting in interdisciplinary teams slightly more often than 
their non-DN counterparts. 

8 For information on all of the components and the percentage of schools to adequately implement each of 
them, see Appendix Table A.3. 
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Strong Learning Environments 

Under the Strong Learning Environments input, each DN school implements a staffing 
model that facilitates both interdisciplinary teacher teams and subject-area professional learning 
communities with common planning time set aside during the school day. Teachers also meet at 
regularly scheduled times to discuss immediate student needs, plan interventions, and coordi- 
nate instructional expectations and practices. The daily school schedule is organized into ex- 
tended class periods (four classes of 70 to 90 minutes each) to allow for in-depth instruction and 
differentiated instructional strategies that meet the needs of students with different learning 
styles. In a DN high school, the school building and students are organized into academies, in- 
cluding a ninth-grade academy and thematic or career academies for tenth- through twelfth- 
graders, each of which houses a moderate-sized group of students. Each academy has dedicated 
administrators and counselors, and an interdisciplinary team of teachers — four to six teachers 
covering the core subjects of math, English/language arts, science, and social studies/history — 
who work with the same students throughout the year. 

The fidelity score for Strong Learning Environments during the second year of imple- 
mentation (0.69) indicates that DN schools implemented a moderate proportion of the compo- 
nents. The average score for the top quartile of schools was 0.85 while the average score for the 
bottom quartile was 0.47, suggesting that schools that implemented more of the model overall 
also tended to be more successful with this input (see Table 2.1). Table 2.2 presents the compo- 
nents where DN schools were most successful (that is, those implemented adequately by at least 
80 percent of schools) and least successful (that is, those implemented adequately by fewer than 
50 percent of schools). As can be seen from the table, four out of the six components fell into 
the middle range, implemented adequately by between 50 and 80 percent of schools. These 
components include creating interdisciplinary teams of teachers working with small academies 
of students, and structuring the school day with extended periods for core academic courses. As 
Table 2.2 indicates, all DN school-based Diplomas Now teams had established norms for col- 
laboration, communication, and decision making among themselves, an important first step in 
implementing other model components. Schools were least successful at scheduling interdisci- 
plinary teacher team meetings. 

Table 2.3 compares DN and non-DN schools in their implementation of programs simi- 
lar to the components of the Strong Learning Environments input. During the second year of 
implementation, core teachers at DN schools were somewhat more likely to report that they 
taught in classes that lasted 70 to 90 minutes than their counterparts in non-DN schools. DN 
teachers also reported that they spent more time collaborating with an interdisciplinary team of 
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Diplomas Now 
Table 2.2 

Pillar I: Teacher Teams and Small Learning Communities, 
Components Adequately Implemented Most and Least Often, Year 2 

Input and Description 


Strong Learning Environments 3 

Number of components 6 

Number of components adequately implemented by 80 percent of DN schools or more 1 

Number of components adequately implemented by 50 percent of DN schools or fewer 1 

Selected component descriptions: (% of schools adequately implementing) 

Site-based team had established standards for collaboration, 
communication, and decision making. 100.0 

Meetings where interdisciplinary core teachers discuss shared students 
occurred multiple times per week. 29.0 


Number of schools 3 1 


SOURCE: Diplomas Now fidelity of implementation program staff surveys, 2012, 2013, and 2014. 

NOTES: For a list of all components and the percentage of schools adequately implementing each 
one, see Appendix Table A. 3. One DN school closed prior to the second year of implementation and 
is not included in this analysis. 

a Strong Learning Environments includes components specific to either middle schools or high 
schools. 


teachers that shared the same students than teachers at non-DN schools . 9 However, on average, 
the amount of time that DN teachers reported working in interdisciplinary teams was less than 
one hour per week, which is below the DN model goal of multiple times per week. 

The fidelity of implementation and service contrast findings for the Strong Learning 
Environments input generally appear to align with each other, and these results suggest that DN 
schools were not only moderately successful in implementing the components of this input, but 
that this level of implementation led to differences between DN and non-DN schools. 


9 Results from surveys of administrators support these findings (see Appendix Table B.2). 
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Diplomas Now 
Table 2.3 


Pillar I: Teacher Teams and Small Learning Communities, 
Service Contrast, Teacher Responses, Year 2 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P- Value 

Strong Learning Environments 






Core teachers reported how many of their classes 
were 70 to 90 minutes. (0 = none, 5 = few, 10 = most) 

6.4 

5.2 

1.2 

0.24 *** 

<0.001 

Core teachers reported the frequency they 
collaborated with an interdisciplinary team of 
teachers who shared the same group of students. 
(0 = never, 5 = less than weekly, 10 = daily) 

5.7 

4.4 

1.3 

0.40 *** 

<0.001 

Core teachers reported hours spent each week 
collaborating with an interdisciplinary team and 
teaching coordinated content across disciplines. 
(0 = none, 5 = 1 to 2, 10 = 3 or more) 

3.9 

3.1 

0.8 

0 29 *** 

<0.001 

Math and English/language arts teachers reported 
participating in a professional learning community 
with teachers within the same subject area. 

(0 = never, 5 = less than weekly, 10 = daily) 

5.4 

5.1 

0.4 

0.13 

0.285 

Sample size 

31 

30 





SOURCE: Follow-up surveys of teachers administered during the school years of 2012-2013 and 
2013-2014. Respondents included middle school teachers who taught sixth or seventh grade and high 
school teachers who taught ninth or tenth grade. 

NOTES: "Core" academic areas are identified as math. English/language arts, sciences, and social 
studies. Across 61 study schools, 1,269 core teachers participated in the follow-up survey, including 
827 math and English/language arts teachers. One DN school closed prior to the second year of 
implementation and is not included in this analysis. For each of the above measures, data are missing 
for no more than 3.4 percent of the teachers. The difference in the percentage of missing data between 
DN and non-DN schools is no more than 1 .2 percent for any of the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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Pillar II: Curriculum and Instruction with Professional Development 

Two inputs are aligned with this pillar: Professional Development and Peer Coaching 
and Curriculum for College Readiness. Key findings from this section include: 

• Curriculum for College Readiness and Professional Development and Peer 
Coaching had the lowest fidelity scores of any inputs during the second year 
of implementation (0.31 and 0.42, respectively). 

• Curriculum for College Readiness was not considered critical to model im- 
plementation by the model designers, and only 4 of the 24 components in- 
cluded in this input were considered critical, which may have contributed to 
the low score. Some schools may have had similar types of curricular re- 
forms already in place, which also may have contributed. According to the 
school staff survey responses used in the service contrast analysis, both DN 
and non-DN schools had relatively high levels of implementation in this area, 
which suggests that many districts were already using some sort of reform 
curricula and offering transitional support classes for struggling students, alt- 
hough the quality of these services was not measured. 

• Although DN schools struggled to fully implement the teacher coaching 
program, math and English/language arts teachers at DN schools reported 
receiving instructional mentoring and support from coaches and school 
leaders four more times per month than teachers at non-DN schools. This is 
consistent with the fidelity of implementation finding that most math and 
English/language arts teachers received coaching during at least one period 
per week. 

Professional Development and Peer Coaching 

At every DN school, school-based instructional coaches and regional instructional facil- 
itators work with English/language arts and math teachers. Coaches provide initial training for 
all transition courses, instruction in how to teach an extended class period, professional devel- 
opment and classroom support, and customized training and workshops. The goal is to have all 
math and English/language arts teachers participating in a two-week coaching cycle that con- 
sists of instructional coaches working with teachers to plan lessons, teach alongside them, mod- 
el teaching techniques, and debrief instructional practices. Instructional facilitators work with 
schools to customize curricula and align instruction with district standards and local initiatives, 
support coaches and teachers, establish professional learning communities, and identify and 
train experienced teachers to teach alongside less experienced ones, model lessons, serve as cur- 
riculum coaches, and support the full implementation of the Diplomas Now model. 
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Diplomas Now 
Table 2.4 


Pillar II: Curriculum and Instruction with Professional Development, 
Components Adequately Implemented Most and Least Often, Year 2 

Inputs and Descriptions 

Professional Development and Peer Coaching 3 

Number of components 5 

Number of components adequately implemented by 80 percent of DN schools or more 0 

Number of components adequately implemented by 50 percent of DN schools or fewer 2 

Selected component descriptions: (% of schools adequately implementing) 

English/language arts instructional coaches completed an instructional 

coaching cycle with teachers at least every two weeks. 45.2 

Mathematics instructional coaches completed an instructional coaching cycle 

with teachers at least every two weeks. 38.7 

Curriculum for College Readiness 3 

Number of components 24 

Number of components adequately implemented by 80 percent of DN schools or more 1 

Number of components adequately implemented by 50 percent of DN schools or fewer 17 

Selected component descriptions: (% of schools adequately implementing) 

Middle school met high standards of implementation of Common Core standards 

in mathematics. 81.3 

High school implemented the Transitions to Advanced Mathematics 

curriculum for 9th-grade students. 46.7 

High school implemented Geometry Foundations reform curriculum for 

lOth-grade students. 33.3 

High school offered at least 6 units of the freshman seminar to 

9th-grade students. 26.7 

Middle school implemented the Savvy Readers' Lab reform curriculum or 

approved alternative. 25.0 

High school implemented the Reading and Writing in Your Career 

reform curriculum for lOth-grade students. 20.0 

Number of schools 3 1 

(continued) 
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Table 2.4 (continued) 


SOURCE: Diplomas Now fidelity of implementation program staff surveys, 2012, 2013, and 2014. 

NOTES: For a list of all components and the percentage of schools adequately implementing each 
one, see Appendix Table A.3. One DN school closed prior to the second year of implementation and 
is not included in this analysis. 

a Professional Development and Peer Coaching and Curriculum for College Readiness include 
components specific to either middle schools or high schools. 

The fidelity score for Professional Development and Peer Coaching during Year 2 was 
0.42, meaning that DN schools were generally unable to meet the goals for high fidelity of im- 
plementation of this input. The average score for the top quartile of schools was 0.84, while the 
average score for the bottom quartile was 0.05, suggesting that some schools were able to im- 
plement these components quite successfully, but many schools struggled with implementation 
(see Table 2.1). Over 50 percent of schools had both math and English/language arts instruc- 
tional coaches providing some coaching at least once a week, but, as seen in Table 2.4, most 
schools found it difficult to establish a consistent instructional coaching cycle for all mathemat- 
ics and English/language arts teachers that consisted of a coach working with each teacher to 
plan, coteach, and debrief on at least one class every two weeks. Even though DN schools 
struggled to implement this input, Table 2.5 shows that English/language arts and math instruc- 
tors at DN schools still reported receiving instructional coaching four times per month more 
than their counterparts at non-DN schools. 

Curriculum for College Readiness 

To prepare students for high school graduation and college, Diplomas Now provides 
schools with research-based curricula for all core subjects along with accelerated remediation 
courses to help students catch up. Talent Development Secondary’s reform curricula are intend- 
ed to promote active learning, close skill gaps, develop mature thinking, and improve achieve- 
ment. Diplomas Now curricula for middle schools integrate high school readiness and evidence- 
based core academic principles in math, English/language arts, science, and social studies. DN 
middle schools also have access to the Mastering the Middle Grades curriculum, designed to 
facilitate the transition to middle school through instruction in life and study skills. In high 
schools Diplomas Now offers a freshman seminar curriculum to ease the transition into high 
school, and evidence-based college preparatory curricula in core subjects. 

The input-level fidelity score for Curriculum for College Readiness during the second 
year of implementation was 0.3 1 , the lowest of all inputs. Many schools reported having dis- 
trict-wide curricular standards already in place. They were hesitant to change to the curricula 
Diplomas Now specified. Schools could meet the requirements of some of these components by 
using alternative curricula deemed adequate by Diplomas Now, but the low implementation 
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Diplomas Now 
Table 2.5 


Pillar II: Curriculum and Instruction with Professional Development, 
Service Contrast, Teacher Responses, Year 2 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P-Value 

Professional Development and Peer Coachine 

Average times per month math and English/language 
arts teachers reported working with an 
instructional coach. J 

7.7 

5.1 

2.6 0.30 ** 

0.020 

Average times per month math and English/language 
arts teachers reported receiving support from 
a school leader or a coach. a 

13.6 

9.6 

4.0 0.28 ** 

0.024 

Curriculum for College Readiness 

Math and English/language arts teachers reported 
implementing the same college preparation 
curriculum with all students in their classes. 

(high school only) (%) 

62.8 

63.1 

-0.3 -0.01 

0.957 

Math and English/language arts teachers reported 
struggling students in their courses received 
additional classes to catch up with their peers. (%) 

68.7 

70.7 

-2.0 -0.05 

0.665 

Math and English/language arts teachers reported the 
average frequency they used a variety of 
student-centered teaching strategies. b 
(0 = never, 5 = less than weekly, 10 = daily) 

7.3 

7.1 

0.2 0.11 

0.247 

Math and English/language arts teachers reported the 
average frequency students applied classroom 
activities to real-life issues and used critical thinking 
and reasoning skills. (0 = never, 5 = less than weekly, 
10 = daily) 

7.1 

7.1 

0.0 0.00 

0.974 

Math and English/language arts teachers reported the 
average frequency academic courses included career 
applications and exploration, (high school only) 

(0 = never, 5 = less than weekly, 10 = daily) 

4.9 

4.8 

0.1 0.04 

0.795 

Sample size 

31 

30 




(continued) 
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Table 2.5 (continued) 


SOURCE: Follow-up surveys of teachers administered during the school years of 2012-2013 and 2013- 
2014. Respondents included middle school teachers who taught sixth or seventh grade and high school 
teachers who taught ninth or tenth grade. 

NOTES: Across 61 study schools, 827 math and English/language arts teachers participated in the 
follow-up survey. Across 29 study high schools, 465 high school math and English/language arts 
teachers participated in the follow-up survey. One DN school closed prior to the second year of 
implementation and is not included in this analysis. For each of the above measures, data are missing for 
no more than 6.2 percent of the teachers. The difference in the percentage of missing data between DN 
and non-DN schools is no more than 3.7 percent for any of the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a Survey items in this input have been calculated by weighting the frequency per month and then 
summing the weighted survey items in order to get a total times-per-month measure. For example, 0 = 
never, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 12 = 
more than once a week but not daily, and 20 = daily. 

b Student-centered teaching strategies include demonstrations, modeling of strategies, and minilessons. 

scores across schools suggest that most schools were unsuccessful in meeting these goals as set 
by the model developers. Top-quartile schools had an average score of 0.50 on this input and 
bottom-quartile schools had an average score of 0.25, suggesting that all schools had difficulty 
implementing this input (see Table 2.1). This may also be in part because there are 24 compo- 
nents in this input and only 4 of those were considered critical by the model’s designers. As 
seen in Table 2.4, 17 of the 24 curricular components were implemented by fewer than 50 per- 
cent of schools. 

Teachers at DN and non-DN schools reported similarly high levels of implementation 
on measures related to Curriculum for College Readiness, leaving little difference between DN 
and non-DN schools in this area (see Table 2.5). About the same percentage of Eng- 
lish/language arts and math teachers in both DN and non-DN schools — a majority in both cas- 
es — reported that the same college preparatory curriculum was offered to all students in their 
classes and that struggling students were offered additional courses to catch up to their peers. 
This suggests that in teachers’ eyes, many schools were already implementing these types of 
practices encouraged by the Diplomas Now model. 


Pillar III: Tiered Student Supports 

As noted above, Pillar III was one of the most successfully implemented pillars. Three 
inputs align with this pillar: Tiered Intervention Model, Student Supports, and Student Case 
Management. Key findings from this section include: 

• Tiered Intervention Model was among the most fully implemented inputs. A 
large majority of schools reported using data to identify students who fall off 
track and reported that teachers and other staff members collaborated to plan 
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interventions for those students. Teachers at DN schools were also more like- 
ly than teachers at non-DN schools to report these types of activities were 
happening. 

• In the input of Student Supports, most DN schools reported high levels of in- 
class and after-school activities. Teachers at DN schools reported much high- 
er levels of academic support in their classes than those at non-DN schools, 
and sixth- and ninth-grade teachers at DN schools reported a higher frequen- 
cy of academic tutoring and attendance and behavior coaching. 

• Student Case Management was moderately implemented, with the majority 
of schools providing case-managed students with individually tailored plans, 
along with enrichment and motivational activities. DN schools performed ac- 
tivities connected to the case management of at-risk students more frequently 
than non-DN schools. 

Tiered Intervention Model 

Each DN school establishes a tiered intervention model guided by an Early Warning 
Indicator (EWI) system that uses data to trigger interventions targeted to student needs. EWI 
system data provide schools with on- and off-track indicators and early warning flags related to 
student attendance, behavior, and course performance — the ABCs, as the Diplomas Now part- 
ners refer to them. The goal of the EWI system is to ensure that the right students receive the 
right intervention at the right time and at the right intensity. At each school, EWI meetings — 
which include some combination of the Talent Development school transformation facilitator, 
administrators, the interdisciplinary team of teachers who work with the students, City Year 
AmeriCorps members, and Communities In Schools site coordinators — are regularly sched- 
uled to review and analyze EWI data. This EWI team works together to identify students who 
show signs of going off track, decide on the appropriate resources or services for each student, 
and implement the appropriate level of intervention for that student. 

The Tiered Intervention Model input was the second most fully implemented input, 
with a fidelity score of 0.75. Many schools were quite successful with implementation of this 
input and the average score for the top-quartile schools was 0.96, but a few schools were not at 
all successful, bringing down the average score for the bottom quartile to 0.45 (see Table 2.1). 
As shown in Table 2.6, over 80 percent of schools were able to implement all the components 
in this input. These components include successfully integrating a tiered intervention model into 
the school day, having an EWI system in place, and holding EWI meetings at least every two 
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Diplomas Now 
Table 2.6 


Pillar III: Tiered Student Supports, 

Components Adequately Implemented Most and Least Often, Year 2 

Inputs and Descriptions 


Tiered Intervention Model 

Number of components 3 

Number of components adequately implemented by 80 percent of DN schools or more 3 

Number of components adequately implemented by 50 percent of DN schools or fewer 0 

Selected component descriptions: (% of DN schools adequately implementing) 

Coordinated Early Warning Indicator system alerts teachers as students begin 

to demonstrate off-track indicators. 90.3 

Tiered intervention model was integrated into the school day. 83.9 

Interdisciplinary Early Warning Indicator team meetings occurred at least biweekly. 80.6 

Student Supports 

Number of components 24 

Number of components adequately implemented by 80 percent of DN schools or more 8 

Number of components adequately implemented by 50 percent of DN schools or fewer 5 

Selected component descriptions: (% of DN schools adequately implementing) 

Additional in-class support was offered in math/English language arts year-round. 100.0 

City Y ear AmeriCorps members were in math/English language arts classrooms at 

least three times per week. 100.0 

At least four whole-school activities were conducted throughout the year. 100.0 

Students received math tutoring from City Y ear AmeriCorps members at a ratio 

equal to or greater than 7 to 1. 48.4 

At least 75% of math and English/language arts classrooms had City Year 

AmeriCorps members embedded. 41.9 


(continued) 


weeks. As seen in Table 2.7, teachers at DN schools also reported doing these types of activities 
slightly more often than non-DN teachers. Overall, Tiered Intervention Model was among the 
inputs that showed the most measures with statistically significant service contrast between DN 
and non-DN schools. 
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Table 2.6 (continued) 


Inputs and Descriptions 


Student Case Management 

Number of components 1 4 

Number of components adequately implemented by 80 percent of DN schools or more 2 

Number of components adequately implemented by 50 percent of DN schools or fewer 3 

Selected component descriptions: (% ofDN schools adequately implementing) 

At least 75% of case-managed students were provided with individual student plans. 96.8 

Communities In Schools site coordinators provided whole-school 

enrichment/motivation services. 96.8 

Communities In Schools site coordinators provided whole-school professional 
physical health services. 38.7 

Communities In Schools site coordinators provided whole-school 

community-service opportunities. 25.8 

Communities In Schools site coordinators provided whole-school professional 

mental health services. 9.7 


Number of schools 3 1 


SOURCE: Diplomas Now fidelity of implementation program staff surveys, 2012, 2013, and 2014. 

NOTES: For a list of all components and the percentage of schools adequately implementing each one, 
see Appendix Table A. 3. One DN school closed prior to the second year of implementation and is not 
included in this analysis. 


Student Supports 

Students identified by at least one Early Warning Indicator as beginning to fall off track 
are offered Tier II support. These students receive additional services from City Year Arneri- 
Corps members, which can include behavior and attendance coaching or management, after- 
school programs, academic tutoring (both individual and group), one-on-one mentoring, and 
near-peer role modeling. Students also receive in-class support services, such as one-on-one or 
small-group tutoring, from City Year AmeriCorps members usually embedded in their Eng- 
lish/language arts and math classes. 

Student Supports was among the most fully implemented inputs, with a fidelity score of 
0.73. Moreover, top-quartile schools had an average score of 0.78 and bottom-quartile schools 
had an average score of 0.67, suggesting successful implementation of this input across schools 
regardless of their overall implementation level (see Table 2.1). In most schools, City Year 
AmeriCorps members provided in-class support for math and English/language arts classrooms 
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Diplomas Now 
Table 2.7 

Pillar III: Tiered Student Supports, 
Service Contrast, Teacher Responses, Year 2 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P-Value 

Tiered Intervention Model 






Core teachers reported the average frequency 
they used attendance, behavior, and course 
performance data to target at-risk students. 

(0 = never, 5 = less than weekly, 10 = daily) 

5.4 

4.9 

0.5 

0.17 

** 0.046 

Core teachers reported hours spent each week 
reviewing student data with an interdisciplinary team 
of teachers to identify student needs. (0 = never, 

5 = 1 to 2, 10 = 3 or more) 

4.1 

3.3 

0.8 

0.28 

*** 0.003 

Core teachers reported the average frequency 
they participated in meetings with administrators 
or counselors to identify at-risk students and 
plan interventions. (0 = never, 5 = less than weekly, 
10 = daily) 

4.9 

3.3 

1.6 

0.57 

*** <0.001 

Core teachers reported the average frequency 
they invited parents of at-risk students to 
participate in discussions about interventions to 
support their children. (0 = never, 5 = less than 
weekly, 10 = daily) 

4.3 

4.1 

0.2 

0.06 

0.378 

Student Supports 






Math and English/language arts teachers reported 
students who were often disruptive were offered 
behavior coaching. (0 = never, 5 = sometimes, 

10 = always, as needed) 

5.5 

5.2 

0.3 

0.09 

0.273 

Math and English/language arts teachers reported 
the frequency students were provided additional 
support related to attendance. (0 = never, 

5 = less than weekly, 10 = daily) 

6.6 

6.2 

0.5 

0.16 

0.102 

Math and English/language arts teachers reported 
the frequency students were offered individual 
or small-group tutoring during their classes. 

(0 = never, 5 = less than weekly, 10 = daily) 

7.1 

6.9 

0.3 

0.09 

0.326 


(continued) 
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Table 2.7 (continued) 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size P-Value 

Average times per month math and English/ 
language arts teachers reported students received 
whole-class behavioral support from volunteers. 2 

7.8 

4.2 

3.6 

0.52 *** <0.001 

Average times per month math and English/ 
language arts teachers reported students received 
academic help in class from volunteers. 2 

43.8 

21.2 

22.7 

0.73 *** <0.001 

Student Case Management 





Core teachers reported the frequency they arranged 
intensive support and interventions for students 
by making appropriate referrals. (0 = never, 

5 = sometimes, 10 = always, as needed) 

7.1 

6.6 

0.5 

0.19 ** 0.020 

Core teachers reported the frequency needs 
assessments were conducted to quickly identify 
students' needs and address them effectively. 

(0 = never, 5 = sometimes, 10 = always, as needed) 

6.0 

5.6 

0.5 

0.16 ** 0.033 

Core teachers reported the frequency academic 
and nonacademic services were coordinated for 
students at risk of dropping out of school. 

(0 = never, 5 = sometimes, 10 = always, as needed) 

6.3 

5.3 

1.0 

0.34 *** <0.001 

Sample size 

31 

30 




SOURCE: Follow-up surveys of teachers administered during the school years of 2012-2013 and 
2013-2014. Respondents included middle school teachers who taught sixth or seventh grade and high 
school teachers who taught ninth or tenth grade. 

NOTES: "Core" academic areas are identified as math. English/language arts, sciences, and social 
studies. Across 61 study schools, 1,269 core teachers participated in the follow-up survey, including 
827 math and English/language arts teachers. One DN school closed prior to the second year of 
implementation and is not included in this analysis. For each of the above measures, data are missing 
for no more than 6.9 percent of the teachers. The difference in the percentage of missing data between 
DN and non-DN schools is no more than 5.3 percent for any of the above measures. 

A two-tailed t-test is used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 

“Survey items in this input have been calculated by weighting the frequency per month and then 
summing the weighted survey items in order to get a total times-per-month measure. For example, 0 = 
never, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 12 = 
more than once a week but not daily, and 20 = daily. 
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at least three times per week, attendance and behavior coaching for students, at least four whole- 
school activities to encourage and promote positive school climate and culture, and after-school 
programs or extended learning time opportunities for students. But, as shown in Table 2.6, DN 
schools did have difficulty ensuring City Year AmeriCorps members were embedded in 75 per- 
cent of English/language arts classrooms and maintaining a student-to-tutor ratio of 7:1 or less. 

As shown in Table 2.7, math and English/language arts teachers at DN schools reported 
that volunteers provided support in their classes approximately twice as often as teachers at non- 
DN schools. Although as a whole teachers reported that students received behavior and attend- 
ance coaching and individual and small group tutoring at about the same rates at DN and non- 
DN schools, sixth- and ninth-grade teachers at DN schools did report higher frequencies of 
these activities than sixth- and ninth-grade teachers at non-DN schools (see Appendix Table 
B.12). This disparity may in part be due to Diplomas Now’s conscious decision to concentrate 
on the sixth and ninth grades and to ensure that students in these transitional grades receive ser- 
vices in cases where there are limited resources. 

In general, the fidelity and service contrast findings for the Student Supports input seem 
to correspond with each other. Although the model led schools to offer considerably more stu- 
dent support services than would have otherwise been possible, limited resources may have cur- 
tailed the services available to students not in transition years (sixth and ninth grades), where 
City Year AmeriCorps members were concentrated. 

Student Case Management 

Students in need of intensive interventions receive individual case management from 
the Communities In Schools site coordinator. The site coordinator conducts a needs assessment 
to determine the range, scale, and scope of the specialized support these students need. Next, the 
site coordinator develops an individual plan for each student, provides referrals for social ser- 
vices, coordinates individual or group counseling, completes home visits, and develops peer 
support groups. Working daily or weekly with at-risk students, the site coordinator serves as a 
resource for these students and monitors their progress. 

The fidelity score for the Student Case Management input averaged 0.59. Scores for the 
top and bottom quartile of schools were quite similar (0.56 and 0.55 respectively) suggesting 
that success in student case management was not much affected by success in the other inputs 
(see Table 2.1). As shown in Table 2.6, the majority of DN schools provided case-managed stu- 
dents with individual plans, enrichment activities, and motivational services. DN schools strug- 
gled most with providing professional physical health services, community-oriented services, 
and professional mental health services during Year 2. In general, schools were quite successful 
in implementing the components identified as critical to the model; those critical components 
focus on offering case-managed students attendance, behavior, and academic support services. 
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Program components not considered critical to the Diplomas Now model — those focused on 
providing physical and mental health services and community services to the whole school — 
were less likely to be implemented fully (see Appendix Table A.3). This suggests that case 
managers may have focused on implementing the components identified as most critical to the 
model. 


As seen in Table 2.7, DN schools were slightly more likely than non-DN schools to of- 
fer the following services related to the case management of at-risk students: teachers arranging 
for intensive support and interventions for students by making appropriate referrals, needs as- 
sessments being conducted quickly to identify students’ needs and address them effectively, and 
coordination and monitoring occurring of academic and nonacademic services for students at 
risk of dropping out of school. Administrators at DN schools also reported a higher frequency 
than non-DN administrators of student participation in individual, group, or family counseling. 

Pillar IV: Can-Do Culture and Climate 

School reform is difficult, and school staff members often have too much to do when 
they are asked to effect change. The inputs of Integrated On-Site Support and Family and 
Community Involvement are about providing additional support to schools to facilitate the hard 
work of school change. This section includes the following key findings: 

• Integrated On-Site Support had the highest fidelity score of any input, sug- 
gesting that the staff members needed to implement the Diplomas Now mod- 
el were in place. The service contrast findings suggest that this influx of staff 
members to DN schools differed from what happened in non-DN schools. 

• The Family and Community Involvement aspects of the DN model have not 
yet been fully implemented at most DN schools, and there was little differ- 
ence between DN and non-DN schools in this area. 

Integrated On-Site Support 

To carry out all the programmatic, structural, and curricular components of the Diplo- 
mas Now model, numerous support staff members are needed. 

• The Talent Development school transformation facilitator, who works 
with the school’s leadership team to organize Diplomas Now’s efforts and to 
launch a tiered intervention model guided by early warning data 

• The City Year program manager, who manages a team of 8 to 20 City Year 
AmeriCorps members to provide academic assistance, run after-school pro- 
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grams, and operate other programs intended to improve school climate, at- 
tendance, and student behavior 

• The Communities In Schools site coordinator, who organizes whole-school 
activities and provides individual student case management 

• The instructional coaches, who provide instructional and curricular support 
to math and English/language arts teachers 

• The instructional facilitators, who provide technical assistance to instruc- 
tional coaches and school administrators 

These staff members make it possible for schools to plan and implement the Diplomas Now 
model. 


Integrated On-Site Support was the most fully implemented input, with a fidelity score 
of 0.87. Additionally, there was no significant difference between the fidelity scores of top- and 
bottom-quartile schools, suggesting uniform and consistent implementation across schools re- 
gardless of their overall implementation level (see Table 2.1). Moreover, as shown in Table 2.8, 
10 out of 1 1 program components were adequately implemented by 80 percent or more of the 
DN schools. These components include having in place math and English/language arts instruc- 
tional facilitators, Communities In Schools site coordinators, a Talent Development school 
transformation facilitator, and at least one AmeriCorps member for every 49 students who need 
Tier II support. It is also worth noting that technical assistance was provided by Talent Devel- 
opment Secondary to 100 percent of DN schools to support Diplomas Now implementation (see 
Appendix Table A.3). 

As seen in Table 2.9, DN schools were much more likely to have volunteers in Eng- 
lish/language arts and math classes than non-DN schools, likely due to the influx of City Year 
AmeriCorps members at the DN schools. 10 As shown in Appendix Table B.5, administrators at 
DN schools also were more likely than administrators at non-DN schools to report that their 
schools had a designated adult — other than the principal or assistant principal — coordinating 
interventions and resources and overseeing math and English/language arts coaches. 

Family and Community Involvement 

The Diplomas Now model encourages schools to make a concerted effort to engage in 
family and community partnerships that promote student success. As schools develop career 


10 City Year worked in five non-DN schools due to relationships with those schools established prior to 
Diplomas Now implementation. 
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Diplomas Now 
Table 2.8 


Pillar IV: Can-Do Culture and Climate, 

Components Adequately Implemented Most and Least Often, Year 2 

Inputs and Descriptions 


Integrated On-site Support 

Number of components 1 1 

Number of components adequately implemented by 80 percent of DN schools or more 1 0 

Number of components adequately implemented by 50 percent of DN schools or fewer 0 

Selected component descriptions: (% of DN schools adequately implementing) 

Math and English/language arts instructional facilitators provided technical assistance. 100.0 

Communities In Schools site coordinator was at school. 100.0 

At least 75% of City Year Ameri Corps members were retained throughout 

the school year. 93.5 

School transformation facilitator began serving school prior to the first day of school. 87.1 

Family and Community Involvement 

Number of components 6 

Number of components adequately implemented by 80 percent of DN schools or more 2 

Number of components adequately implemented by 50 percent of DN schools or fewer 1 

Selected component descriptions: (% of DN schools adequately implementing) 

School offered workshop to parents at least once during school year. 90.3 

Teachers sent information to parents on how to help their children at least 

once during school year. 83.9 

School formally recruited parents and trained them to work as school volunteers. 41.9 


Number of schools 3 1 


SOURCE: Diplomas Now fidelity of implementation program staff surveys, 2012, 2013, and 2014. 

NOTES: For a list of all components and the percentage of schools adequately implementing each one, 
see Appendix Table A. 3. One DN school closed prior to the second year of implementation and is not 
included in this analysis. 


academies, they involve local businesses, professionals, and government agencies in planning 
and implementing activities and work experiences for students, such as job shadowing and in- 
ternships. City Year AmeriCorps members lead community service opportunities for students 
and conduct outreach to parents. In cooperation with the school’s staff, the Communities In 
Schools site coordinator and City Year AmeriCorps members also organize and participate in 
school events such as health fairs for family and community members. 
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Diplomas Now 
Table 2.9 

Pillar IV: Can-Do Culture and Climate, 
Service Contrast, Teacher Responses, Year 2 


Survey Item 

DN 

Schools 

Non-DN Estimated Effect 
Schools Difference Size 

P-Value 

Integrated On-Site Support 






Average times per month math and English/language 
arts teachers reported City Year AmeriCorps members 
worked with students. 3 

14.4 

1.7 

12.7 

2.15 

*** <0.001 

Average times per month math and English/language 
arts teachers reported City Year AmeriCorps members, 
college students, or volunteers from organized 
programs worked with students. 3 

18.2 

3.4 

14.8 

1.70 

*** <0.001 

Average times per month math and English/language 
arts teachers reported any volunteer worked 
with students. a,b 

23.7 

6.9 

16.8 

1.03 

*** <0.001 

Family and Community Involvement 






Teachers reported the frequency parents and 
community members were offered opportunities to 
participate in school initiatives. (0 = never, 5 = less 
than weekly, 10 = daily) 

4.3 

4.3 

0.1 

0.03 

0.708 

Teachers reported the school had a plan for parent 
and community engagement linked to specific goals 
for improving student learning and healthy 
development. (%) 

54.2 

53.3 

0.9 

0.02 

0.769 

Teachers reported the school helped all parents 
understand what they could do at home to support 
a student's success in school. 

(0 = strongly disagree, 10 = strongly agree) 

6.3 

6.2 

0.1 

0.04 

0.561 

Sample size 

31 

30 





(continued) 


The fidelity score for the Family and Community Involvement input averaged 0.50. The 
average score for top-quartile schools was 0.70 and average score for bottom-quartile schools 
was 0.35, suggesting that schools struggling with implementation in general were also less suc- 
cessful with this input (see Table 2.1). Still, as shown in Table 2.8, over 80 percent of schools 
met adequacy requirements for two of the six components by sponsoring at least one workshop 
for parents and sending home infomiation to parents about how to help their children with 
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Table 2.9 (continued) 


SOURCE: Follow-up surveys of teachers administered during the school years of 2012-2013 and 2013- 
2014. Respondents included middle school teachers who taught sixth or seventh grade and high school 
teachers who taught ninth or tenth grade. 

NOTES: Across 61 study schools, 1,818 teachers participated in the follow-up survey, including 827 
math and English/language arts teachers. One DN school closed prior to the second year of 
implementation and is not included in this analysis. For each of the above measures, data are missing for 
no more than 7.8 percent of the teachers. The difference in the percentage of missing data between DN 
and non-DN schools is no more than 3 percent for any of the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a Survey items in this input have been calculated by weighting the frequency per month and then 
summing the weighted survey items in order to get a total times-per-month measure. For example, 0 = 
never, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 12 = more 
than once a week but not daily, and 20 = daily. 

b Volunteers include: City Year AmeriCorps members, students from local college or 
teacher-preparation programs, members of organized volunteer groups, peer tutors, and community 
members. 


schoolwork at least once during the year. Still, most schools were unable to formally recruit and 
train parents to act as school volunteers. It is worth noting that this input is not considered criti- 
cal to Diplomas Now model implementation and only one of the six components was consid- 
ered critical. 

Teachers in DN and non-DN schools were asked whether their school had a plan for 
parent and community engagement, the frequency with which parents were offered opportuni- 
ties to participate in school initiatives, and the degree to which the school helped parents under- 
stand how to support students’ academic success. As shown in Table 2.9, teachers reported 
some communication and engagement with parents and community members, and there were 
only negligible differences between DN and non-DN schools for any of these measures. 

The somewhat low fidelity score and lack of service contrast on this input is not alto- 
gether surprising. Increasing the engagement of families is a particularly difficult challenge for 
middle and high schools, since parents tend to lessen their involvement in their children’s 
schooling as children grow older. 1 1 


n Mulhall, Mertens, and Flowers (2001); Juvonen et al. (2004). 
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Summary 

In all, DN schools were moderately successful in meeting the Diplomas Now model’s imple- 
mentation goals, and service contrast findings suggest that Diplomas Now-type activities were 
more prevalent in DN schools than in non-DN schools. DN schools implemented the majority 
of the program components in Teacher Teams and Small Learning Communities, Tiered Stu- 
dent Supports, and Can-Do Culture and Climate (Pillars I, III, and IV, respectively), and find- 
ings from the service contrast analysis suggest that the Diplomas Now model was improving 
programs in these areas relative to non-DN schools. The model’s goals for involving parents 
and community members in school activities and decision making were not fully realized in 
Year 2, but a large majority of schools did meet an adequate level of implementation for some 
of the components. Development and training of the school-level staff members from each of 
the three partner organizations was implemented with fidelity to the model and these staff 
members reported high levels of access to each other. 

Fidelity scores were lowest for Curriculum and Instruction with Professional Develop- 
ment (Pillar II). Service contrast findings reveal, however, that teachers at both DN and non-DN 
schools reported using the types of curricula and related teaching strategies specified by the Di- 
plomas Now model, which suggests that similar reforms may have already been taking place at 
the schools (although it cannot be measured how closely aligned these alternative curricula were 
with those of the Diplomas Now model). Although DN schools were unable to reach the im- 
plementation goals for peer coaching and professional development set by the Diplomas Now 
model, instructional coaching for teachers was happening at DN schools at higher levels than at 
non-DN schools. 

The inputs of Professional Development and Peer Coaching, Tiered Intervention Mod- 
el, and Strong Learning Environments saw the biggest differences in fidelity between top- and 
bottom-quartile schools, suggesting that what set those top-quartile schools apart was their 
greater ability to implement school-wide refonns that required changes in district or school pol- 
icies and structures. 

Given the complexity and volume of activities they had to undertake, the level of im- 
plementation achieved by DN schools seems reasonable for their second year with the model, 
though there is still room for them to grow and improve in subsequent years. While the model’s 
designers expected that DN schools would adequately implement the majority of inputs and 
components, this study represents the first time the Diplomas Now model has been measured in 
this way, and therefore it remains uncertain what minimum thresholds of implementation fideli- 
ty are needed to make a difference to schools and students. Still, these implementation fidelity 
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and service contrast findings may have implications for the Diplomas Now impact analysis, 
which will be discussed in the next report. A program is more likely to have an impact on out- 
comes if it provides services that differ (for example, in quality, nature, or frequency) from the 
services students would have otherwise received. 

The next chapter provides a deeper look into the fidelity and service contrast findings to 
provide insights into the variation in implementation from Year 1 to Year 2. 
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Chapter 3 


Consistency and Change in Implementation 
Between Year 1 and Year 2 


As discussed in the previous chapter, Year 2 fidelity of implementation data suggest that DN 
schools implemented the majority of program components, and that doing so led to notable dif- 
ferences from non-DN schools. Still, there was room for improvement: The overall average im- 
plementation score was 0.62 for the second year (on a scale of 0 to 1), quite similar to the first- 
year score. 1 Given the complexity of the Diplomas Now model, it is not surprising that DN 
schools could not fully implement all components of the model in the first year, and it might be 
expected that implementation would intensify over the first several years. On the other hand, the 
Diplomas Now model is an accelerated system of reform, bringing many resources to a school 
including a large number of auxiliary staff members. It might be expected that the most change 
would happen in the first year of implementation, while work in subsequent years would focus 
on maintaining and deepening strategies already implemented. 

It is also worth noting that many of the Diplomas Now model inputs expand into 
schools a grade level at a time. During the first year of implementation, Diplomas Now empha- 
sized services to the sixth grade in middle schools and ninth grade in high schools. In Year 2, 
Diplomas Now continued to offer support to this group of students as they transitioned into the 
seventh and tenth grades, but also made it available to the incoming cohort of sixth- and ninth- 
grade students. In other words, even though little change is seen in the implementation score 
from Year 1 to Year 2, the Year 2 score represents services provided to many more students. 

This chapter compares Year 1 and Year 2 implementation in more depth, looking at 
both the fidelity of implementation and service contrast measures to illuminate whether imple- 
mentation changed in specific areas over the two years, and if so, in what ways. The main find- 
ings discussed in this chapter include: 

• The overall fidelity score remained consistent from Year 1 to Year 2, but 
scores improved in Integrated On-Site Support and Program Staff Training 
and Development, reflecting improvements in the appointment and training 
of Diplomas Now staff members. 

• Although fidelity of implementation changed little from Year 1 to Year 2, the 
contrast between DN schools and non-DN schools increased in several areas, 


'The overall average fidelity of implementation score in Year 1 was 0.61. 
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including the collaboration of teachers in interdisciplinary teams, the profes- 
sional development of teachers through peer coaching, and the use of data to 
identify struggling students. This contrast suggests that Diplomas Now 
brought more consistency and stability to the DN schools than they would 
have experienced otherwise. 

The following sections explore how the fidelity scores and the differences in services 
provided by DN and non-DN schools (or “service contrast”) changed from the first to the sec- 
ond year of implementation. 


Comparison of Year 1 and Year 2 Fidelity Scores 

Fidelity scores generally remained consistent from the first to the second year. Although DN 
schools implemented most components in both years, on average little additional growth oc- 
curred in the second year. 

Table 3.1 provides average fidelity scores for each year overall, and by pillar and input. 
Although small fluctuations occurred across pillar and input fidelity scores, only Integrated On- 
Site Support and Program Staff Training and Development demonstrated statistically significant 
increases in fidelity to the model from the first to the second year. Further analysis of fidelity 
scores revealed that DN schools demonstrated several statistically significant improvements at 
the level of the components in these inputs: (1) Both the Talent Development school transfor- 
mation facilitator and Communities In Schools site coordinator were hired and on site before the 
school year began, (2) the school-based team attended the three-day Diplomas Now summer 
institute, (3) the school transformation facilitator completed the five-day summer training ses- 
sion, and (4) a joint planning meeting took place prior to the start of the school year. 2 

Although implementation scores for the Student Supports input stayed consistent over 
the two years, the addition of the seventh and tenth grades meant that the number of students 
supported under this input grew quite a bit in most schools during the second year. The lower 
score in the second year for the Student Case Management input may be due in part to that same 
expansion, which caused larger caseloads for Communities In Schools case managers. While 
the percentage of schools meeting the model’s goals for case management and academic sup- 
port went up in Year 2, other support services seemed to have suffered to ensure those goals 
were met (for example, life skills/social development services and mental health services at 


“See Appendix Table A.3 for detailed information on the percentage of DN schools adequately imple- 
menting each component. 
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Diplomas Now 
Table 3.1 


Fidelity of Implementation Findings, All DN Schools, Comparison of Year 1 and Year 2 


Model Inputs 

Year 1 

Average Fidelity Score 
Year 2 

Difference 

Pillar I. Teacher Teams and Small Learning Communities 

0.67 

0.69 

0.02 

Strong Learning Environments a,b 

0.67 

0.69 

0.02 

Pillar II. Curriculum and Instruction with Professional Development 

0.37 

0.36 

-0.01 

Professional Development and Peer Coaching d b 

0.44 

0.42 

-0.02 

Curriculum for College Readiness b 

0.30 

0.31 

0.01 

Pillar III. Tiered Student SuDDorts 

0.72 

0.69 

-0.03 

Tiered Intervention Model 3 

0.76 

0.75 

0.00 

Student Supports 3 

0.72 

0.73 

0.00 

Student Case Management 3 

0.69 

0.59 

-0.10 *** 

Pillar IV. Can-Do Culture and Climate 

0.65 

0.69 

0.04 

Integrated On-Site Support 3 

0.82 

0.87 

0.05 ** 

Family and Community Involvement 

0.48 

0.51 

0.03 

Program Staff Training and Development 

0.61 

0.71 

0.10 *** 

Overall 

0.61 

0.62 

0.01 

Sample size 

32 

31 



(continued) 




Table 3.1 (continued) 

SOURCE: Diplomas Now fidelity of implementation program staff surveys, 2012, 2013, and 2014. 

NOTES: Each of the nine inputs consists of a set of more specific components measured on a 0-1 
scale. For each DN school, all of the component scores under an input are averaged to create the 
implementation score for that input. These scores are then averaged to create the pillar-level and 
overall implementation scores. 

One DN school closed prior to the second year of implementation and is not included in the Year 
2 analysis. 

A two tailed t-test was applied to differences. Statistical significance levels are indicted as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

indicates model inputs designated as critical to the Diplomas Now model. 
b Strong Learning Environments, Professional Development and Peer Coaching, and Curriculum 
for College Readiness include some components specific to either middle schools or high schools. 


some schools). The level of implementation for three of the five components in this input identi- 
fied as critical increased from Year 1 to Year 2, while the level of implementation for all but 
one of the components not identified as critical decreased, suggesting that schools focused on 
those components deemed to be critical (see Appendix Table A.3). It is also possible the case 
managers at the schools made decisions about what was most important to implement that were 
slightly different from the originally specified program goals, adapting to the needs of specific 
schools and students. For example, the percentage of schools meeting the goals for behavior and 
attendance support services declined in the second year, but Communities In Schools case man- 
agers may have decided to focus on these services less since City Year AmeriCorps members 
were also supposed to concentrate on them (under the Student Supports input). 

Figure 3.1 provides a visual comparison of school-level fidelity scores by pillar for 
Year 1 and Year 2. For each pillar, each DN school’s Year 1 fidelity score was plotted against 
its Year 2 score, providing a snapshot of the overall trend in perfonnance. Schools plotted 
above the diagonal experienced an increase in implementation during Year 2, while those 
schools below the line experienced a decline. For all pillars, the overall trend suggests consistent 
implementation, on average, from the first to the second year. Moreover, the figure supports the 
implementation findings discussed in Chapter 2 that schools had greater success in implement- 
ing Teacher Teams and Small Learning Communities, Tiered Student Supports, and Can-Do 
Culture and Climate (Pillars I, III, and IV) than Curriculum and Instruction with Professional 
Development (Pillar II). These findings are consistent across the two years. 


Comparison of Year 1 and Year 2 Service Contrast 

This section investigates whether differences between DN and non-DN schools in the imple- 
mentation of Diplomas Now-like practices grew or shrank in the second year of implementa- 
tion. In other words, the section investigates whether the service contrast between DN and non- 
DN schools changed over time. This comparison is based on surveys of teachers who taught 
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Year 2 implementation score Year 2 implementation score 


Diplomas Now 
Figure 3.1 


Comparison of Year 1 and Year 2 Fidelity Scores for Each DN School, by Pillar 


Pillar I: Teacher Teams and Small Learning Environments Pillar II: Curriculum and Instruction with Professional Development 



0 0.2 0.4 0 ; 6 0.8 1 * 0 0.2 0.4 0.6 0.8 1 

Year 1 implementation score Year 1 implementation score 



0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 

Year 1 implementation score Year 1 implementation score 

(continued) 



Figure 3.1 (continued) 

SOURCE: Diplomas Now fidelity of implementation program staff surveys, 2012, 2013, and 2014. 

NOTE: Each data point represents the Year 1 and Year 2 pillar-level fidelity score for an individual 
DN school. One DN school closed prior to the second year of implementation and is not included in 
the Y ear 2 analysis. 

sixth- and ninth-grade students during Year 1 and sixth-, seventh-, ninth-, and tenth-grade stu- 
dents during Year 2. The surveys cover different grades in Year 1 and Year 2 so as to capture 
the teachers most likely to participate in program implementation during each year. It should be 
noted that because these service contrast findings are based on only two years of data, there is a 
fairly significant degree of uncertainty involved in drawing conclusions about trends. Still, ex- 
ploring the differences and similarities between Year 1 and Year 2 service contrasts offers a 
sense of how Diplomas Now implementation has changed the trajectory of DN schools. 

Figure 3.2 presents a graphical comparison of the contrast between DN and non-DN 
schools across the first two years of implementation. Each data point in these scatter plots repre- 
sents teachers’ average Year 1 response to a specific survey question under each pillar plotted 
against the average Year 2 response. The fact that most of the data points are above the line 
suggests that the contrast in school practices between DN and non-DN schools was consistently 
greater in Year 2 than Year l. 3 Twenty- seven of the 29 school practices measured in the teacher 
surveys had a greater service contrast in Year 2 than in Year 1. The shaded data points represent 
survey items that did not show a statistically significant difference between DN and non-DN 
schools in Year 1, but did show a statistically significant difference by Year 2. Although some 
of the measures only show a minimal increase across implementation years, 9 of the survey 
items across the first three pillars show statistically significant differences between DN and non- 
DN schools in Year 2 but not in Year 1 . 4 

For example, within Pillar I, there was a statistically significant difference between DN 
and non-DN schools in the frequency with which teachers collaborated in interdisciplinary 


3 For these graphical comparisons, the teacher survey measure results are drawn from Appendix Tables 
B.6-B.9 for Year 1 and Tables 2.3, 2.5, 2.7, and 2.9 for Year 2. 

4 Regardless of statistical significance during a particular year, 12 of the measures indicate significant 
growth in service contrast between implementation years. A variation of the impact model presented in Ap- 
pendix B was used to test whether there were statistically significant differences between the service contrast 
estimates in Year 1 and those in Year 2. Data from both implementation years were included in the impact 
model. For each survey measure, the p-value of the difference between the Year 1 and Year 2 service contrast 
estimates was obtained by interacting the implementation-year variable with each parameter and running a 
three-level model (that is, teachers within school-year clusters, and school year within school). 
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Year 2 effect sizes Year 2 effect sizes 


Diplomas Now 
Figure 3.2 

Comparison of Year 1 and Year 2 Service Contrast Teacher Measures, by Pillar 


Pillar I: Teacher Teams and Small Learning Environments 



Year 1 effect sizes 
Pillar III: Tiered Student Supports 



Year 1 effect sizes 
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Figure 3.2 (continued) 


SOURCE: Follow-up surveys of teachers administered during the school years of 201 1-2012, 
2012-2013, and 2013-2014. 

NOTES: Each data point represents the difference (in effect size) between DN and non-DN 
teachers' responses to each survey item for Year 1 and Year 2. Data points that are shaded 
represent survey items that were not statistically significant at the 10 percent level in Year 1 but 
became statistically significant in Year 2. Unshaded data points represent survey items that were 
either statistically significant in both years or not statistically significant in either year. A two- 
tailed t-test was used for all statistical tests. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for 
the respective measures. 


teacher teams in Year 2, but not in Year l. 5 In Pillar II, math and English/language arts teachers 
at DN schools did not report receiving professional development and peer coaching significant- 
ly more often than non-DN teachers during Year 1, but did report receiving it significantly more 
often during Year 2. 6 While there had been statistically significant contrast between DN and 
non-DN schools for only 3 of the 12 measures in Pillar III during Year 1, 5 additional practices 
associated with Tiered Student Supports showed statistically significant differences during Year 
2, including teachers’ use of data to target struggling students and all 3 of the practices related to 
the case management of students most at risk of dropping out. 7 In Pillar IV, all of the Integrated 
On-Site Support measures show contrast in both years while none of the Family and Communi- 
ty Involvement measures show much contrast in either year. 8 

Since service contrast is a relative measure of implementation, changes in its magnitude 
can be driven by fluctuations in the services available at non-DN schools as well as DN schools. 
For example, as noted above, there is a statistically significant difference between DN and non- 
DN schools in the frequency with which teachers collaborated in interdisciplinary teacher teams 
in Year 2, but not in Year 1. This could be because teachers at DN schools collaborated with 
interdisciplinary teams more frequently in Year 2, or because teachers at non-DN schools col- 
laborated with interdisciplinary teams less frequently. 

Figure 3.3 demonstrates visually whether the increases in service contrast from Year 1 
to Year 2 were because DN schools improved, non-DN schools declined, or a combination of 
both. These graphs suggest that it was mostly a combination of both. DN schools generally im- 
proved their implementation from the first to the second year. However, this improvement in 


5 See Appendix Table B.6 for Year 1 teacher survey results for Pillar 1 and Table 2.3 for Year 2. 

6 See Appendix Table B.7 for Year 1 teacher survey results for Pillar II and Table 2.5 for Year 2. 

7 See Appendix Table B.8 for Y ear 1 teacher survey results for Pillar III and Table 2.7 for Year 2. 

8 See Appendix Table B.9 for Year 1 teacher survey results for Pillar IV and Table 2.9 for Year 2. 
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Diplomas Now 
Figure 3.3 

Differences Between DN and Non-DN Schools in Changes in School Structures and Practices 

from Year 1 to Year 2, by Pillar 

Pillar I: Teacher Teams and Small Learning Environments Pillar II: Curriculum and Instruction with Professional Development 
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Figure 3.3 (continued) 


SOURCE: Follow-up surveys of teachers administered during the school years of 201 1-2012, 
2012-2013, and 2013-2014. 

NOTE: All values are expressed in terms of standard deviations (SD). Standard deviations have 
been estimated using the sample of non-DN teachers in Y ear 1 for sixth and ninth grades. 


DN schools was not as great as the amount that non-DN schools declined in their implementa- 
tion of similar activities. 

For example, school practices such as block schedules and teachers collaborating in in- 
terdisciplinary teams (Pillar I) and a tiered system of interventions (Pillar III) showed increased 
implementation at DN schools from Year 1 to Year 2, but an increase smaller than the corre- 
sponding decline in implementation at non-DN schools. In Pillar II, teachers at DN schools ac- 
tually reported their schools were slightly less successful in implementing elements of teacher 
professional development and college readiness curricula in Year 2 than they were in Year 1. 
Nevertheless, the contrast in services between DN and non-DN schools still grew because 
teachers at non-DN schools reported stronger decreases associated with these inputs. Across the 
two years of implementation, the increase at DN schools in additional on-site staff members and 
engagement of families and communities (Pillar IV) was approximately equivalent to the drop 
in these areas at non-DN schools. 

Overall, two key findings emerge from the comparison of service contrasts between 
Year 1 and Year 2. First, the differences between DN and non-DN schools became greater in 
Year 2 than they were in Year 1 . Second, these increases in service contrast appear to be due to 
a combination of DN schools improving their implementation over time and non-DN schools 
reducing their implementation of similar practices. Thus, not only does the service contrast sug- 
gest that Diplomas Now improves the implementation of key practices at DN schools, it also 
suggests that Diplomas Now may be preventing these schools from suffering a decrease in the 
implementation of key practices that they would have experienced otherwise. 


Comparison of Fidelity of Implementation and Service Contrast 

Both fidelity of implementation and service contrast showed gains in measures associated with 
Integrated On-Site Support between Year 1 and Year 2. But in general, fidelity showed little 
growth between Year 1 and Year 2 for most pillars, while there was some growth in service 
contrast across all pillars. There are at least two main factors that could explain these differences 
between the service contrast and fidelity findings. 
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First, the two analyses use different data sources. Surveys of Diplomas Now staff mem- 
bers supplied most of the data for the fidelity analysis while surveys of teachers and administra- 
tors supplied the data for the service contrast analysis. It may have taken teachers at DN schools 
more time to fully engage with and understand the components of the Diplomas Now model. As 
a result, from the perspective of teachers Diplomas Now may have intensified between the first 
and second year of implementation even though Diplomas Now staff members did not see 
much difference. This theory is supported by the fact that sixth- and ninth-grade teachers — 
who were more likely to have participated in the program for two years — reported even 
stronger service contrast during the second year than the full sample of sixth-, seventh-, ninth-, 
and tenth-grade teachers. Sixth- and ninth-grade teachers may have had a deeper understanding 
of the program in Year 2 than seventh- and tenth-grade teachers, who may have been encounter- 
ing it for the first time. 9 

Second, the service contrast accounts for structures and school practices implemented in 
both DN and non-DN schools, whereas the fidelity analysis only examines DN schools’ pro- 
grams. Consequently, the decline at non-DN schools from Year 1 to Year 2 in Diplomas Now- 
like practices contributed significantly to the increase in service contrast, but did not affect the 
fidelity results. 


Summary 

Comparisons of the first and second year demonstrate clear improvements in DN schools’ im- 
plementation of the Integrated On-Site Support and Program Staff Training and Development 
inputs, suggesting that Diplomas Now was better equipped by the second year to hire and ade- 
quately train staff members in a timely fashion. Still, between Year 1 and Year 2, there is little 
overall difference in the fidelity of implementation across the rest of the inputs. As noted in the 
last chapter, it is not yet known what level of implementation is needed to effect change at 
schools so it is hard to judge whether this lack of growth in implementation from Year 1 to Year 
2 is a problem. Given that implementation of the Diplomas Now model is accelerated by design 
— with all components rolled out in the first year and an influx of additional staff members de- 
ployed to move refonns forward quickly and offer extra support to students — it may not be sur- 
prising that little change occurs in Year 2. It could be that the majority of the change Diplomas 
Now effects in a school occurs by the end of Year 1. It is also possible that the lack of change in 
the fidelity score can be attributed to the additional grade levels being supported in Year 2. 
Schools may have been extending additional effort, but spreading that effort over larger numbers 
of students. Finally, it is also worth noting that the fidelity score generally measures whether pro- 


9 See Appendix Tables B.10-B.13 for service contrast findings during Year 2 for sixth- and ninth-grade 
teachers only. 
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gram components are being implemented at all, and if they are, at what frequency. It does not 
measure the quality of implementation. If program components were implemented better in Year 
2 — as opposed to simply more — that would not be captured by the fidelity score. 

In contrast to the fidelity score, service contrast measures showed quite a bit of growth 
from Year 1 to Year 2. This disparity may reflect the difference in the perspectives and experi- 
ences of the survey respondents who provided the data for the two analyses, Diplomas Now 
staff members for the fidelity score and teachers and administrators for the service contrast 
measures. The consistency in program implementation over the two years at DN schools also 
seems to contrast with a lack of consistency at non-DN schools, suggesting that Diplomas Now 
may have brought stability to DN schools. 

Drawing on qualitative data from selected DN schools, Chapter 4 expands on the con- 
tinued challenges and successes of Year 2 implementation. It explores how DN staff members 
deal with external factors, such as principal turnover, that may impede successful implementa- 
tion, and also touches on effective practices for embedding Diplomas Now in schools and sus- 
taining it in the long term. 
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Chapter 4 


Emerging Successes and Continued Challenges 
in Year 2 Implementation 


This chapter examines Year 2 qualitative data collected at a set of case study Diplomas Now 
schools (DN schools) in several districts. It expands on themes identified in the previous report 
on first-year implementation, which concluded with a discussion of continued challenges and 
examples of ways to address implementation barriers. Grounding these findings in the existing 
research literature on school reform serves as a way to make these data more meaningful and 
build upon the current understanding of the complexities of improving schools. The findings in 
this chapter supplement the fidelity of implementation and service contrast findings discussed in 
the previous two chapters by exploring the interview and focus group data collected in the case 
study schools. It offers a more in-depth look at the accomplishments of Diplomas Now staff 
members and the challenges they faced in implementing the model. Given that these data are 
drawn from a subset of the Diplomas Now sample, caution is advised in generalizing findings to 
all DN schools. 

Previous research into comprehensive school reform efforts has identified factors that 
contribute to sustained school change. One such factor is the ability to find and maintain an ap- 
propriate balance between fidelity to the reform model and adaptation to the local context. The 
research literature also notes the importance of aligning the planned school reform activities 
with existing school practices and requirements. Additionally, successful implementation hinges 
on teacher, administrator, and district support for and ownership of the reform. The extent to 
which teachers, principals, and policies remain constant (that is, the extent to which the refonn 
environment remains stable) is also an important influence on implementation. 1 Similarly, 
schools that sustain reforms often demonstrate continuity of leadership, commitment among 
stakeholders, and integration of practices into the school structure and culture. Turbulent district 
conditions, on the other hand, can make it difficult for reforms to endure. 2 Taken together, these 
concepts inform the interpretation of four central themes from the second year of Diplomas 
Now implementation: 

• Various factors external to the program (such as school closures, principal 
turnover, and budget cuts) affected the implementation of Diplomas Now. 

The findings offer examples of how to adapt to changing school contexts. 

'Desimone (2002). 

2 Datnow (2005). 
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• Data from Year 2 implementation pointed to progress in Diplomas Now be- 
coming “part of the school.” In many cases, Diplomas Now staff members 
better understood how to align the program with school goals and gamer 
support from school staff members. 

• Diplomas Now staff turnover presents both challenges and opportunities in 
continuing the momentum of program implementation. 

• Sustaining program components at the end of the Investing in Innovation (i3) 
grant period is not yet a primary concern for many school or program staff 
members who participated in case studies; however, participants shared some 
early thinking about this issue. Participants recognized the need to maintain 
the services offered by additional Diplomas Now personnel and the im- 
portance of gaining support from administrators to make a priority of ele- 
ments of Diplomas Now (such as arranging schedules to include common 
planning time). 

This chapter presents findings derived primarily from the analysis of qualitative data 
collected through interviews and focus groups at nine Diplomas Now case study schools in four 
school districts. 3 Year 2 fidelity scores at case study schools were similar to the scores of the 
overall sample (0.63 and 0.62, respectively; detailed case study implementation scores by year, 
pillar, and input can be found in Appendix Table C.2). Individuals from the case study schools 
shared their perceptions of the second year of Diplomas Now implementation in response to 
questions about the following topics: (1) collaboration, (2) factors that facilitated and hindered 
implementation, (3) sustainability, and (4) lessons and recommendations. Data analysis includ- 
ed systematic coding of interview and focus group transcripts and followed best practices in 
qualitative research (see the discussion of methodology in Appendix C). 

The sections that follow present (1) contextual factors external to Diplomas Now, (2) 
the integration of the Diplomas Now model into the fabric of the school, (3) the implications of 
turnover within the Diplomas Now team, and (4) early lessons about program sustainability. 
Each section includes a description of implementation challenges in Year 2 with references to 
how these barriers were similar or different to those schools faced in Year 1. Each section also 
includes a discussion of successes and the factors that facilitated them as well as recommenda- 
tions for implementing Diplomas Now and other whole-school reforms. The chapter includes 


3 The analysis is based on all data collected from case study schools in Year 2 of Diplomas Now im- 
plementation. It includes data collected in spring 2013 from one school that closed after Year 2 as well as 
data collected in spring 2014 from a replacement case study school from the same district, also in its sec- 
ond year of Diplomas Now implementation. 
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examples to illustrate common themes that emerged across participants and schools and con- 
cludes with an interpretation of key findings. 


The Influence of Factors External to Diplomas Now 

“In education, the only thing consistent is change.” This field manager’s insight echoes early 
findings from Year 1 presented in the first Diplomas Now report, which described how chang- 
es in the school that are external to Diplomas Now could impede the implementation of the 
program. Analysis of the qualitative data indicates that external challenges related to school 
staff turnover and reduced school resources persisted during the second year of implementation 
at some case study schools. Added challenges in Year 2 related to district decisions to close or 
reconstitute schools, replace administrators, and lay off school staff members. Such factors 
were beyond the control of the Diplomas Now partners and presented great challenges to im- 
plementation, and school support for program activities often waxed or waned because of 
them. Some case study schools maintained momentum while others no longer had the support 
to continue certain Diplomas Now activities. This dynamic mirrors the pattern seen among DN 
schools as a whole, as illustrated in Figure 3.1, which depicts the variation in school fidelity 
scores across years. 

The Diplomas Now model intentionally targets at-risk schools, typically those eligible 
for Title I funds, 4 primarily in urban school districts with large populations of low-income and 
minority students. Its designers anticipated that contextual factors like limited resources, varia- 
ble and inconsistent staffing, and contentious school climates would affect both DN and non- 
DN schools. Although contextual factors like these posed challenges to the implementation of 
the Diplomas Now model, the consistency and stability provided by Diplomas Now probably 
mitigated their effect. As discussed in Chapter 3, the implementation of Diplomas Now-like 
practices in non-DN schools declined from the first year to the second year of implementation, 
which may reflect the adverse effect of these contextual factors. 

The sections that follow illustrate various scenarios at case study schools facing closure, 
principal turnover, and budget cuts. The examples also showcase the ways that Diplomas 
Now’s presence mitigated the effects of these difficult transitions. 


4 Title I funds from the U.S. Department of Education go to schools with high numbers or high percentages 
of students from low-income families. 
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School Closures and Transitions 


Four out of nine case study schools from three districts were slated to close (two), near- 
ly closed (one), 5 or were reconstituted to serve additional grades (one) during the second year of 
implementation. The four schools included two middle schools and two high schools. Among 
the 3 1 DN schools in the study in Year 2, 5 were slated to close by Year 3. 6 The experiences at 
case study schools differed. Schools that overcame these problems tended to cite administrator 
or district support for the Diplomas Now model and the flexibility of Diplomas Now staff 
members in adapting their roles or practices. 

At one of these schools that was slated to close, the administrator, the district, and a 
cohesive Diplomas Now leadership team (Talent Development school transfomiation facilita- 
tor, Communities In Schools site coordinator, and City Year program manager) strongly sup- 
ported the Diplomas Now model and reportedly worked well together. All these factors helped 
sustain strong implementation through the second year even while teachers and students were 
acclimating to the many changes taking place at their school, and dealing with new uncertain- 
ties stemming from its pending closure. The district also required common planning time, 
which facilitated the implementation of other Diplomas Now activities like Early Warning In- 
dicator (EWI) meetings. Although implementation remained mostly intact during the “phas- 
ing-out” process, students reported that the transition negatively affected school climate and 
student morale. However, one teacher observed that the school climate had actually improved, 
and attributed that to the presence of Diplomas Now in the building. Reportedly, Diplomas 
Now staff members offered support that counteracted some aspects of the negative school cli- 
mate related to the transition. 

In contrast, another school facing a major transition lacked district support for Diplomas 
Now, particularly during the second year of implementation, when the district leader who had 
championed Diplomas Now left the district. According to the field manager for this district, 
“Nobody really wanted to take responsibility on the district’s end to be the guardian of Diplo- 
mas Now.... The person who actually signed the letter of intent is no longer here.” As a result, 
the school transformation facilitator no longer had access to the district-level data necessary for 
EWI meetings, despite having had access in Year 1 . 

One case study school offered an example of Diplomas Now staff members supporting 
the school through an otherwise rough transition. An administrator described how Diplomas 


5 The district decided to close the school, then reversed that decision a few weeks before the school year 
began (the summer before Year 2 of Diplomas Now implementation) and rehired school staff members and 
administrators. 

6 There were originally 32 DN schools in the study, but one school did close after the first year of imple- 
mentation. Of the 30 non-DN schools in the study, 2 were slated to close during Year 2. 
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Now “was a very great partner during that time.... [It] really helped support the teachers through 
these critical times.” Program staff members described how they shifted their stance from trying 
to prevent school closure (by showing data to the district that demonstrated their successes) to 
supporting students and school staff members during the difficult transition. The role that Di- 
plomas Now staff members played in supporting academy structures (including a ninth-grade 
academy) appeared to preserve a sense of community and a focus on the current needs of stu- 
dents. 


Another Diplomas Now team faced many challenges trying to establish relationships 
with the school’s staff and gamer support for activities such as EWI meetings in the midst of the 
school’s transition. According to the school transformation facilitator, “It’s not like [teachers] 
will let you have their full attention to give them information on what EWI meetings should 
look like, if they are not even seeing the same students consistently, and they’re still trying to 
get that squared away.... It just didn’t feel like there was a good, trusting environment.” The 
program manager at this school also reported that the Diplomas Now leadership team spent a lot 
of time at the beginning of the school year trying to rebuild relationships with school staff 
members and administrators. As one focus group participant noted, “Diplomas Now really 
needs to figure out a way to change the way we go about managing our relationships. And that 
also means that, when a school partner gets phased out or when a school is going to close, that 
we don’t just shut the door on the people that we’ve worked with for an entire year.” 

Principal Turnover 

Principal turnover at several case study schools affected implementation in Year 2. 7 A 
third of the case study schools began the second year of Diplomas Now implementation with 
new principals. One of those schools received a second new principal at the beginning of Year 
3. Principal transitions were not specific to case study schools alone and appear to affect other 
DN and non-DN schools as well. Across all 3 1 DN schools in the study, approximately a third 
faced principal turnover during the second year of Diplomas Now implementation. 8 The effect 
of principal turnover at case study schools varied, either bolstering or constraining implementa- 
tion, depending on the new principal’s awareness of and support for Diplomas Now. To over- 
come potential challenges to implementation introduced by a change in leadership, Diplomas 
Now staff members sought to establish relationships with incoming principals early on and 
demonstrated flexibility in their roles to accommodate those new principals’ needs. 


7 Although schools also experienced other administrative changes, a school’s principal serves as its prima- 
ry administrator, so transition in that role is likely to have the most drastic effect on Diplomas Now implemen- 
tation. 

8 The same was true across the 30 non-DN schools in the study: Approximately a third faced principal 
transitions during the second year of Diplomas Now implementation. 
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Program staff members from two case study schools that had new principals expressed 
frustration at having devoted extensive effort to building relationships and securing support for 
Diplomas Now, and then having to “start all over again.” A Diplomas Now regional staff mem- 
ber described it this way: “A lot of it is about making sure that the leadership knows exactly 
what we’re doing and building that trusting relationship.... We’re a partner for the school, so just 
having to reset that every time is a little frustrating.” School and program staff members from 
one school noted that the loss of the principal had a negative effect on students and the school’s 
climate because many students viewed the former principal as a “father figure.” Case study par- 
ticipants reported an increase in fights, a decrease in student motivation, and considerable stu- 
dent frustration about having a new principal. However, school staff members also noted that 
the presence and consistency of Diplomas Now leaders had a strong and positive influence at 
the school. For example, a school counselor stated, “I see a really cohesive team that works well 
together to keep the flow going by trying to keep students on task.... The administrative changes 
have been extremely difficult for the staff, morale- wise, but I think for the students, they’ve got 
their Diplomas Now family.” 

Adversarial relationships between administrators and Diplomas Now staff members can 
cripple implementation. Case study participants spoke about many challenges at one school, 
stemming from a new principal who actively opposed Diplomas Now at the beginning of the 
school year. Nearly all participants at this school attributed the waning implementation of Di- 
plomas Now to a lack of administrator support. The principal’s negative sentiments and frequent 
references to Diplomas Now as an “outside agency” starkly contrasted to Year 1 implementation 
at the school. One teacher described how the Diplomas Now team and the school staff were “like 
family last year.... It was a whole different feeling of the school.” The principal subsequently par- 
ticipated in additional Diplomas Now training and said that it had led to a better understanding of 
the program’s value and sparked a new commitment to ensure it succeeded. 

Budget Cuts and Workforce Reductions 

Diplomas Now is operating in a time when school budget cuts have become common- 
place, affecting school districts across the country. Based on the most recent data available, the 
national median expenditures per pupil for regular school districts decreased 1.5 percentage 
points between 2010 and 201 1; it is expected this trend has continued since these data were last 
reported. 9 Case study participants from four case study schools in two districts cited budget cuts 
that led to subsequent teaching staff reductions as challenges to Year 2 implementation of Di- 
plomas Now. Schools from one district were unable to pay for instructional coaches and did not 


9 Comman, Keaton, and Glander (2013). 
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have sufficient staff time for common planning sessions or EWI meetings in Year 2, practices 
that had been in place during Year 1. A program manager said: 

It’s been a huge challenge with the climate of the school district and the lack of 
resources and the lack of money, not having common planning time with teach- 
ers. We’re not implementing what needs to be implemented.... We don’t have 
anybody to staff an in-school suspension, so we’re just suspending students out 
of school.... Our suspensions have been through the roof. So our [progress to- 
ward the] goal of getting students to have less suspensions is mediocre. 

However, one administrator committed to reestablishing common planning in Year 3, after no- 
ticing students slipping in Year 2 because the school was no longer looking at student data 
through the EWI lens. 

One unintended consequence of budget cuts and layoffs was increased support for Di- 
plomas Now. School staff members from two case study schools in different districts reported 
that they welcomed the additional staff and resources the program provided to schools. For ex- 
ample, veteran teachers at one school reportedly did not understand the benefits of Diplomas 
Now and held negative attitudes toward it in Year 1. After approximately a third of teachers 
were laid off, leading to larger classroom sizes with more than 40 students per class, teachers 
began to express gratitude for Diplomas Now’s contributions of added support inside and out- 
side of the classroom. 


Embedding Diplomas Now in Schools 

Case study data illuminate effective practices that can help embed Diplomas Now staff mem- 
bers and structures into school culture and fabric: (1) aligning program goals with school priori- 
ties and (2) securing administrator and teacher support for the Diplomas Now model. Tying 
these two notions together, one case study participant noted that support is a “two-way street” 
(that is, not only does the school have to support the model, but the Diplomas Now staff has to 
support the school’s goals). However, as discussed later in this chapter, in some cases program 
staff turnover can make it harder for school administrators to sustain support for the Diplomas 
Now model or integrate it into their schools. 

Aligning Goals 

As discussed in Chapter 3, DN schools as a whole demonstrated an improved ability in 
Year 2 to successfully implement Integrated On-Site Support, the Diplomas Now model input 
concerned with providing staff members to a school to help coordinate school transformation, 
staff support, and additional services. Similarly, in Year 2 school and program staff members at 
the case study schools more often perceived Diplomas Now as “part of the school” than they 
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did in Year 1, when greater tensions were reported between program and school staff members. 
In some cases, however, aligning program goals with school priorities remained difficult. Em- 
bedding an external program into a school’s culture seems to require both connecting Diplomas 
Now to the school vision and coordinating among the Diplomas Now partners. 

Diplomas Now staff members and school administrators need to work closely in order 
to align their goals and create shared expectations. Some administrators used the resources that 
came with Diplomas Now — such as additional staff members and EWI data — to pursue their 
own school priorities alongside those of Diplomas Now. Administrators offered the following 
recommendations for other DN schools (also important considerations for any school reform 
initiative): (1) Identify how the program’s staff can help address your school’s unique needs, (2) 
develop the roles of program staff members to fit your approach, and (3) establish procedures 
that keep all stakeholders accountable for their commitments. Case study participants described 
two practices as particularly useful: needs assessments conducted by the Communities In 
Schools site coordinator (in an effort to understand the school context) and City Year Ameri- 
Corps members’ ability to fulfill coordinator roles (such as attendance coordinator). 

Securing Support from School Staff Members 

Another important aspect of embedding Diplomas Now in schools is gamering support 
from administrators and teachers. Much as they did after the first year of implementation, a ma- 
jority of case study participants identified administrator and teacher support as a driving force 
for successful program implementation. Although it remained a challenge to ensure that teach- 
ers and administrators knew about and supported Diplomas Now, there was evidence of im- 
provement along those lines in Year 2, suggesting that school staff members were becoming 
more familiar with the Diplomas Now model. 

At some case study schools a lack of administrator support presented continuing chal- 
lenges to implementation in Year 2, while at other case study schools administrators became 
more enthusiastic in Year 2 because they better understood the mission and staff roles of Di- 
plomas Now. Similarly, case study respondents observed that teachers became more accepting 
of Diplomas Now in Year 2, although securing their active support often remained a challenge. 
Some teachers were reportedly not “fully bought into the program,” resisting activities such as 
instructional coaching, but willing to engage in other elements of Diplomas Now, such as EWI 
meetings. Among other lingering concerns, teachers and administrators said they hesitated 
about Diplomas Now activities like instructional coaching due to reluctance to give up current 
practices, worries that Diplomas Now staff members were evaluating them, and lack of 
knowledge about Diplomas Now. Identifying ways to encourage teacher acceptance of the 
model is particularly important given that Curriculum and Instruction with Professional Devel- 


64 



opment (Pillar II) had the lowest average fidelity score in Year 2 across all schools (as described 
in Chapter 2). 

Diplomas Now staff members identified various approaches they used in Year 2 to in- 
crease teacher and administrator understanding of the program, such as developing resource 
maps to clarify roles and services (including both Diplomas Now and other school personnel 
and programs), demonstrating EWI meetings, and conducting skits to portray a coaching cycle. 
These approaches may have helped more schools implement all the critical components of the 
Strong Learning Environments input. Participants also spoke about the importance of backing 
from the district, both in providing sufficient training opportunities to acquaint school staff 
members with the program and in setting a tone of support for Diplomas Now that leaders at the 
school could emulate. 

Case study participants also spoke extensively about the interaction between teachers 
and City Year AmeriCorps members, commenting on teachers’ increased willingness to call 
upon this resource in their classrooms. As was the case in the first year of implementation, Year 
2 case study data reveal mixed findings about teachers setting clear expectations for City Year 
AmeriCorps members. At one school, City Year AmeriCorps team leaders reported that Ameri- 
Corps member and teacher interactions were much smoother in Year 2, especially with teachers 
already familiar with Diplomas Now from the first year of implementation. Teachers at another 
school reported that they continued to grapple with how to work with City Year AmeriCorps 
members “the way we are supposed to.” As one instructional facilitator said: “What needs to be 
stepped up is the messaging from administration and from us, the model providers, that ... 
teachers need to set aside time purposefully at least once a week to sit down with your assigned 
[AmerijCorps member to review what the content is and specifically what you need their role to 
be in the classroom in supporting you.” Other case study participants offered suggestions for 
clearly defining AmeriCorps member roles that echoed this sentiment. 


Implications of Turnover Within the Diplomas Now Team 

As discussed in previous chapters, the most fully implemented model input was Integrated On- 
Site Support, which covers the on-site support and staffing Diplomas Now provides. That find- 
ing is supported by the service contrast analysis, which shows that DN schools were more likely 
to have volunteers in English/language arts and math classes than non-DN schools. Moreover 
Integrated On-Site Support and Program Staff Training and Development demonstrated statisti- 
cally significant increases in implementation from the first to the second year. Some schools 
continued to face challenges in these areas, however, and the case study data illuminate some of 
the nuances of their experiences. 
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During the first two years of implementation, all nine case study schools experienced 
turnover in the Diplomas Now leadership team, as one or more of its members left. This turno- 
ver came in addition to the expected turnover of City Year AmeriCorps members, who typically 
serve one-year temis. The timing of these transitions (whether they occurred between Year 1 
and Year 2 or during the school year) and the specific positions that became vacant varied 
across schools. Such changes may be inevitable, and some case study schools handled them 
better than others. 

Many case study participants reported that this staff turnover increased the workload of 
returning team members, as they took on responsibilities not part of their original job descrip- 
tions. For example, one Communities In Schools site coordinator reported working with EWI 
data instead of student case management. Similarly, another school relied heavily on the City 
Year program manager after the school transfonnation facilitator from Talent Development 
Secondary departed. 

In some cases, the departure of a key Diplomas Now team member also slowed pro- 
gram momentum. For example, the site coordinator at one school started to establish relation- 
ships with students, but then departed in the middle of Year 2. Other Diplomas Now staff mem- 
bers at the school could not deliver the same intensive Tier III services (such as counseling or 
home visits) because, as they noted in focus groups, they had neither the qualifications nor the 
experience necessary. In lieu of hiring a new site coordinator, a staff member from Communi- 
ties In Schools came to the school once a week to help students adjust. However, “kids have 
problems five days a week, not just [on] Thursday,” as a City Year AmeriCorps team leader 
said, and many students reportedly felt slighted and abandoned. At another school, two staff 
members from the Diplomas Now leadership team and a school administrator all departed after 
the first year of Diplomas Now implementation. A City Year AmeriCorps member at this 
school spoke about how these changes made it difficult to sustain the work they had accom- 
plished in Year 1: “I don’t think we [the Diplomas Now team] all sat down and talked about 
how to reconnect the dots to provide the full range of services.” 

There were also instances, however, in which incoming and existing staff members 
worked together effectively to make transitions seamless and sustain Diplomas Now implemen- 
tation. It helped when schools were able to maintain at least some consistency in the Diplomas 
Now leadership team: It preserved useful institutional knowledge about the successes and chal- 
lenges of the first year of Diplomas Now implementation that could be shared with new pro- 
gram staff members. For example, returning Diplomas Now staff members at many schools 
helped to orient new team members and used the lessons they’d learned from Year 1 to “start 
Year 2 off right.” In other cases, City Year AmeriCorps members who became team leaders in 
Year 2 spoke positively about having increased responsibilities after gaining their initial experi- 
ence in Year 1. 


66 



Finally, on occasion staffing changes even brought new opportunities. For example, at 
one school, participants credited a new school transformation facilitator — described as very 
engaged and organized — as the reason that teachers became more receptive to Diplomas Now 
in Year 2. 


Early Lessons to Sustain Diplomas Now 

After the i3 grant currently supporting Diplomas Now ends, the model’s designers envision the 
program sustaining itself in schools in two ways: (1) Communities In Schools site coordinators 
and City Year AmeriCorps members must demonstrate their efficacy well enough that school 
districts, local philanthropies, or other entities are willing to invest in their services; and (2) Di- 
plomas Now must build schools’ institutional capabilities so that they can take on the responsi- 
bilities of school transformation facilitators. 10 Although interview protocols included specific 
questions about sustainability, case study participants offered limited insights about plans for 
continuing elements of the program at the conclusion of the i3 grant period. For the most part, 
Diplomas Now and school staff members were primarily focused on the current and upcoming 
school years. However, many — including both Diplomas Now and school staff members — 
described their concern about what would happen when the entire array of Diplomas Now sup- 
port services were no longer available (at least not in their current form). Talking with partici- 
pants about sustainability often caused them to feel overwhelmed, since most had not begun to 
think about or plan for sustaining program activities beyond the current funding period. 

A few issues seemed to make planning for sustainability particularly challenging. Many 
case study participants observed that the design of the model encourages the school to rely on 
the program. A City Year program manager discussed the thin line between encouraging self- 
sufficiency and helping students and teachers too much: “I feel like we need to be in a position 
where we’re weaning off our support.” City Year AmeriCorps members at another school ex- 
pressed concern about how well the school would function without Diplomas Now. 

The other major barrier to sustainability that case study participants identified was turn- 
over among both school and the Diplomas Now staff members. These transitions often result in 
“reinventing the wheel every time [a new staff person comes on board],” as one program man- 
ager commented. In other words, if the way to make the program sustainable is by building 
schools’ own capabilities, then constant change in key roles makes that much more complex. 


l0 Once DN schools move to “sustaining status,” principals continue to have access to forms of support 
such as summer training, virtual platforms for peer sharing, data reports, and some technical assistance through 
the Diplomas Now Principals Network. 
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In some cases, Diplomas Now and school staff members did make progress towards 
sustainability. Two school transformation facilitators in different districts talked about how they 
trained teachers to take over the facilitation of EWI meetings. Some school administrators re- 
ported taking other steps towards sustainability by making it a priority to put certain elements of 
Diplomas Now in place. Because some aspects of Diplomas Now, like common planning time, 
are not resource-intensive, once administrators see their value they are likely to be motivated to 
incorporate them into existing practices, a step critical to sustainability. This theme is reinforced 
by the service contrast data described in Chapter 2, which indicate that teachers at DN schools 
were more likely than their counterparts at non-DN schools to report collaborating in interdisci- 
plinary teams. 

When case study participants were invited to recommend how their school and other 
schools might sustain the successes achieved by Diplomas Now, common responses included 
additional funding, additional staff members, and support from school staff members for the 
Diplomas Now model. Most participants pointed to funding as the most important factor in con- 
tinuing at least some components of Diplomas Now. As a school transformation facilitator said, 
“Time and money seem to be the biggest things for sustainability.” Aside from funding, other 
important factors for sustaining the program included the contribution of “people power.” Ad- 
ministrators often said that if they had a limited budget and could continue only a few aspects of 
Diplomas Now, they would make a priority of funding for City Year AmeriCorps members. 
Participants also said it would be important to maintain support from school staff members — 
the leaders who champion elements of the program. One City Year program manager com- 
mented it was important for Diplomas Now staff members to foster the notion that the school 
owned the work: “I think sustainability, number one, stems from everyone knowing what their 
role is and knowing how to best engage each partner to leverage the program.” 


Summary 

The findings presented throughout this chapter illustrate how contextual issues, support from 
school staff members, and staff turnover within the program can influence the implementation 
and sustainability of Diplomas Now. The chapter has offered examples of the ways Diplomas 
Now and school staff members achieved successes in the midst of circumstances often out of 
their control. One theme that arose in many of the conversations was the importance of gamer- 
ing support from administrators. None of these case study schools yet epitomizes ideal model 
implementation, in which the Diplomas Now staff found the right balance of flexibility and ad- 
herence to the model’s prescriptions while cultivating ample support from administrators. How- 
ever, taken together, the findings suggest that the mix of those three elements — flexibility, fi- 
delity, and administrative support — may be the key to achieving the goals of Diplomas Now. 
Perhaps having a strong Diplomas Now team enables the various partner organizations to be 
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nimble in response to emerging school needs in a way that stays true to Diplomas Now princi- 
ples. Statistically significant increases from Year 1 to Year 2 in the implementation of Integrat- 
ed On-Site Support and Program Staff Training and Development provide evidence that DN 
schools have made strides in this direction, as do the corresponding findings of increased ser- 
vice contrast in Year 2. It is likely that these improvements will help Diplomas Now teams be- 
come more cohesive and responsive, and encourage administrators to redouble their efforts with 
greater enthusiasm. 
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Chapter 5 

Conclusion 


Diplomas Now is a school reform model combining the efforts of three national organizations 
— Talent Development Secondary, City Year, and Communities In Schools — to work with 
urban schools in low-income communities. It seeks to refine schools’ structures to provide 
more personalized experiences for students, promotes curricula focused on college readiness, 
sets up an Early Warning Indicator data system to identify struggling students, and brings in 
additional staff members and resources to ensure those students receive the extra academic or 
social support they need to be successful. It is a complex, multidimensional program of 1 1 1 
separate program components categorized into nine inputs, being implemented in urban public 
schools that are by nature already complex and constantly changing, each with its own systems 
and needs. The Diplomas Now team at a school works to mold the model’s programs to fit 
those specific needs. 

In the first year, schools implementing Diplomas Now (DN schools) put in place a sub- 
stantial amount of the program model. This level of implementation was sustained during Year 
2, leading to a clear contrast in services between DN and non-DN schools in many areas. The 
contrast in services grew from the first to the second year, suggesting that while implementation 
fidelity (as described in surveys by Diplomas Now staff members at the DN schools) remained 
steady during the first two years, the differences in services between DN and non-DN schools 
(as described in surveys by teachers and administrators) grew. This may suggest that school 
staff members in DN schools came to understand and participate in the program components 
more, and possibly that they became more committed to these reforms. But a decline in similar 
programs and services at non-DN schools also suggests that in an environment of continual 
change and in many cases financial stress, Diplomas Now may have offered stability and con- 
sistent support to DN schools. The following sections highlight some key findings of this report. 


Implementation Fidelity 

During Year 2, DN schools were moderately successful in implementing the model as designed, 
with a fidelity score of 0.62 across all DN schools on a scale from 0 to 1 (where 1 would indicate 
ideal implementation). This level of implementation was similar to that of the first year, even as 
the program expanded into the seventh and tenth grades. The fidelity score compares actual im- 
plementation with the ideal version laid out by the model’s developers. The level of implementa- 
tion needed to elicit positive change has yet to be empirically tested, so it is not yet possible to 
assess whether this level of implementation is enough to have an impact on students’ outcomes. 
It is possible that the lack of growth in the fidelity score from Year 1 to Year 2 reflects stagnant 
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implementation that could be detrimental to the goals of the program. It is also possible that the 
level of implementation brought on by the influx of staff members and resources into the DN 
schools in Year 1 represents the momentum needed to produce the desired reforms, and that 
maintaining this level of implementation in Year 2 is enough to allow the program to progress 
and deepen its effect on schools, teachers and administrators, and ultimately, students. 

Schools were best at employing and training the proper staff, implementing the tiered 
intervention model, and identifying and supporting students who were falling off track to gradu- 
ation. All were areas where the considerable support of Diplomas Now staff members and their 
additional resources likely helped facilitate implementation. Improved fidelity from Year 1 to 
Year 2 for the model inputs of Integrated On-Site Support and Program Staff Training and De- 
velopment suggests an improvement in the appointment and training of Diplomas Now staff 
members in the second year. This shows how the Diplomas Now partners responded to prob- 
lems from the first year that were clearly under their control and ensured that these components 
were better implemented in the second. 

DN schools were least successful in meeting the model’s goals in the Curriculum for 
College Readiness input, which included implementing the Talent Development reform curricu- 
la in English/language arts and mathematics courses, offering Talent Development’s accelerated 
remediation courses for students struggling in those subjects, and implementing curricula to 
support students’ transition into middle or high school. In many cases, participating districts had 
some curricular reforms already in place, and DN schools may have been reticent to change 
from district-supported or -mandated curricula to the curricula promoted by Diplomas Now. 

DN schools also struggled to meet the model’s goals in the Professional Development 
and Peer Coaching input. Most schools were not able to maintain a consistent coaching cycle 
for all mathematics and English/language arts teachers throughout the year. It was probably dif- 
ficult in some cases for districts to provide the necessary instructional coaches in a time of 
shrinking school budgets. It may also be that this level of interaction represents too great a 
change in practice for teachers. 

In general, the findings suggest that program implementation was weakest in areas 
where schools felt they did not need the additional support Diplomas Now offered (curricular 
components), or where teachers needed to change their practices and where additional resources 
were needed to support the change (peer coaching). 

Meanwhile, the biggest differences between schools that implemented the most of the 
model overall and those that implemented the least could be seen in the inputs Professional De- 
velopment and Peer Coaching, Tiered Intervention Model, and Strong Learning Environments. 
These were all areas that required changes in district or school policies and structures, suggest- 
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ing that what separated higher- from lower-implementing schools were these factors not fully 
under the control of the Diplomas Now staff. 


Service Contrast 

As noted earlier, it is unknown what level of model implementation is necessary to elicit positive 
change in schools. One way to better understand what change is happening at the schools is by 
measuring service contrast, which compares the programs and services at DN schools with those 
at similar non-DN schools. Given that schools were randomly assigned to either implement the 
Diplomas Now model or not, the non-DN schools are the strongest representation of what would 
have happened at the DN schools had they not implemented Diplomas Now. Although differ- 
ences between DN and non-DN schools were not always large (except in a couple of areas), they 
were present in many areas, suggesting that Diplomas Now is making a difference in DN 
schools. This difference in implementation between DN and non-DN schools is a promising 
finding: It means that the Diplomas Now model could have an impact on the student outcomes of 
attendance, behavior, and course perfomiance that will be addressed in a future report. 

Meanwhile, although DN schools’ overall average fidelity score did not change from 
Year 1 to Year 2, the service contrast between DN and non-DN schools widened in several are- 
as: the frequency with which teachers collaborated in interdisciplinary teams, the professional 
development of teachers through peer coaching, and the use of data to identify struggling stu- 
dents. This growth in service contrast is at least partly attributable to a decline in the Diplomas 
Now-like services available at non-DN schools, which suggests that Diplomas Now brought 
more stability to the DN schools than they would have had otherwise. 

Since service contrast was measured using teacher and administrator surveys while fi- 
delity of implementation was measured using Diplomas Now staff surveys, it may also be that 
teachers and administrators grew more understanding and accepting of the program from the 
first to the second years even as Diplomas Now staff members were reporting similar levels of 
implementation. In Year 2, those teachers most likely to have been involved in program imple- 
mentation during both years (the sixth- and ninth-grade teachers) reported stronger service con- 
trast in several areas, especially the Student Supports input, than the full sample of all participat- 
ing teachers. 

There was no service contrast between DN and non-DN schools in their implementa- 
tion of a college readiness curriculum or in their levels of parent and community involvement, 
but that is largely because the responses of teachers indicated that district-wide efforts in these 
areas were under way in both DN and non-DN schools. Teachers at all schools reported that 
college readiness curricula and remediation courses for students struggling with math and Eng- 
lish/language arts were in relatively wide use. Staff members at some DN schools were wary to 
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change district- wide or existing curricula at their schools to the Diplomas Now curricula. Activ- 
ities to boost parent and community involvement may take time to implement successfully and 
may have not been the highest priority for program staff members during the early phases of 
implementation. 

It is worth noting that other types of interventions or programs not similar to those be- 
ing planned and implemented in DN schools could be taking place in non-DN schools. Those 
types of interventions or programs would not be captured in the service contrast analyses, which 
are focused solely on the types of refonns included in the Diplomas Now model. Yet it is possi- 
ble that such changes could also affect the student outcomes to be discussed in the next report. 


Case Study Findings 

The Diplomas Now model intentionally targets at-risk schools, typically those eligible for Title 
I funds, 1 primarily in urban school districts with large populations of low-income and minority 
students. It was anticipated that contextual factors like limited resources, variable and incon- 
sistent staffing, and contentious school climates would affect both DN and non-DN schools. 
Although factors like these posed challenges to the implementation of the Diplomas Now mod- 
el, the consistency and stability provided by Diplomas Now probably mitigated their impact. In 
some cases, being flexible and adaptable to the particular needs of the school may be a more 
effective tactic for the Diplomas Now staff than trying to meet every goal of the program mod- 
el. The increasing service contrast between Year 1 and Year 2 described in Chapter 3 lends cre- 
dence to the notion that in challenging circumstances Diplomas Now is enabling schools to 
maintain and increase the implementation of practices essential to continued school improve- 
ment. These lessons can inform the implementation of Diplomas Now moving forward and 
suggest how similar programs might balance flexibility in responding to the needs of schools 
with fidelity to the model as designed. 

In Year 2, interviewees and focus group participants from case study schools reported 
that teachers and administrators were more likely to perceive Diplomas Now as “part of the 
school” than they were in Year 1. This seems to be thanks to the efforts of program staff mem- 
bers to align Diplomas Now model goals with school goals and to gamer stronger support from 
teachers and administrators. These findings comport well with the theory that service contrast 
grew between Year 1 and Year 2 because teachers and administrators began to appreciate the 
benefits of the Diplomas Now model more as they came to understand it better. Still, even 
though Diplomas Now deployed and trained staff members more successfully in Year 2 than in 


'Title I funds from the U.S. Department of Education go to schools with high numbers or high percentages 
of students from low-income families. 
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Year 1, the case study data reveal ongoing challenges related to Diplomas Now staff turnover 
and the integration of new program staff members. The flexibility created by the close commu- 
nication and contact of staff members from the three organizations allowed the Diplomas Now 
school-level teams to retain smoother implementation amid staff turnover than they may have 
otherwise. 

Sustaining the program beyond the current grants underwriting it was a distant concern 
for most Diplomas Now and school staff members, and staff turnover along with schools’ reli- 
ance on Diplomas Now staff members to implement the model’s refonns made it hard for them 
to envision a clear path to sustainability. Still, schools were making progress toward sustaining 
some of the less resource-intensive elements of the program, such as the small learning commu- 
nities structure. Case study participants’ recommendations for sustaining the program included 
gamering administrator support and identifying funds for key program elements such as the 
City Year AmeriCorps members. 


Next Steps 

The first two reports have presented information about the implementation of Diplomas Now, 
setting the stage to evaluate whether the model has an impact on schools and students. Thus far, 
implementation of the model has been moderately successful and has resulted in DN schools 
becoming different from non-DN schools in their structure and organization, and in the practic- 
es employed by their staffs. Over two years, this contrast has increased despite stable implemen- 
tation fidelity, and case study data suggest that at some schools the model is beginning to be- 
come part of the fabric of the school. The next report will present findings about whether this 
effort helps students maintain better attendance, behave better in school, and improve their aca- 
demic performance in core courses. Measures of early school and student outcomes will also 
allow the evaluation team to test assumptions in the Diplomas Now model’s theory of change, 
which may lead to a better understanding of how and in what sequence an integrated refonn 
model like Diplomas Now effects change in a school. 
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Appendix A 

Supplementary Materials for Fidelity 
of Implementation Analyses 




Data Sources 

The Diplomas Now program staff surveys serve as the basis for determining fidelity and 
calculating the fidelity metrics. The Diplomas Now program staff surveys were developed in 
collaboration with the Diplomas Now Implementation Support Team and specifically target the 
components within each input to measure the fidelity of their implementation. The following 
four surveys were administered to program staff members from each of the Diplomas Now 
partner organizations during each study year: 

1. The Implementation Support Team was asked about each school’s participa- 
tion in the Diplomas Now summer institute, in-service training for program 
staff members, and the kick-off planning sessions and meetings. 

2. The school transfonnation facilitator, who coordinates school refonn efforts, 
answered questions regarding student learning environments, the school cur- 
riculum, tiered intervention support, family and community involvement, and 
the presence and activities of curricular support staff members, such as the 
instructional coaches. 

3. The City Year program manager, who manages the City Year AmeriCorps 
members at a school, was asked about their staffing and activities, such as tu- 
toring schedules and after-school programs. 

4. The Communities In Schools site coordinator, responsible for intensive in- 
terventions, answered questions about the presence, start time, and certifica- 
tion of the site coordinator, and about collaboration with Diplomas Now pro- 
gram partners. 


Response Counts 

Appendix Table A.l provides response counts for each of the program staff surveys. Year 1 
program staff surveys were administered in spring/summer 2012 for “wave 1” schools (schools 
that joined the study in the 2011-2012 school year) and spring/summer 2013 for “wave 2” 
schools (schools that joined the study in the 2012-2013 school year). Year 2 program staff 
surveys were administered in spring/summer 2013 for wave 1 schools and spring/summer 2014 
for wave 2 schools. 
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Diplomas Now 
Appendix Table A.l 


Program Staff Survey Response Counts, Year 1 and Year 2 


Year 1 Year 2 

Total Total Total Total 

Respondents Schools Respondents Schools 

Respondent type 





Diplomas Now Implementation Support Team 

32 

32 

31 

31 

Talent Development school transformation facilitator 

32 

32 

31 

31 

City Year program manager 

32 

32 

31 

31 

Communities In Schools site coordinator 

32 

32 

31 

31 

Sample size 

32 

32 

31 

31 


SOURCE: Diplomas Now fidelity of implementation program staff surveys administered in 2012, 2013, 
and 2014. 

NOTE: One DN school closed prior to the second year of implementation and is not included in any of the 
implementation analyses. 


Methodology 

Each of the nine inputs consists of a set of more specific components identified by the Imple- 
mentation Support Team. Given the complexity of the Diplomas Now model and school-wide 
reform, a total of 1 1 1 components were identified under the nine Diplomas Now inputs, based 
on their importance in the context of the logic model as assessed by the Implementation Support 
Team. The fidelity of implementation score reflects all inputs and components of the Diplomas 
Now model. For each component, the evaluation team and the Implementation Support Team 
designed a scale to determine adequate implementation fidelity. A score ranging from 0 to 1 
was calculated for each input, based on its components. (Since original fidelity scales and 
criteria for components varied, components were standardized to a 0-to-l scale.) The average of 
the scores for all inputs is a school’s fidelity score. A school’s fidelity score therefore represents 
the proportion of Diplomas Now components implemented there, and provides flexibility in that 
it credits schools for partially implemented components. The fidelity scores for all schools were 
then averaged to produce pillar-level fidelity scores and an overall fidelity score. 

To illustrate fidelity calculations, Appendix Table A.2 presents an excerpt from the fi- 
delity matrix, highlighting the input Strong Learning Environments and its high school compo- 
nents, the scales and criteria for assessing adequate fidelity, and sample responses from a 
hypothetical school. One can calculate a fidelity score for Strong Learning Environments using 
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Diplomas Now 
Appendix Table A.2 


Example of a Diplomas Now Program Model Input, Related Components, and Fidelity Measures: 

Strong Learning Environments 


Component 

Operational Definition 

Fidelity Scale 3 

Criterion 
for Adequacy 

Sample Response for 
Hypothetical School 

Small learning 
communities 

Interdisciplinary teams of teachers who 
work with the same small group of 
students 

0: No 
1: Yes 

1 : Adequate 

1: Yes 

Interdisciplinary 
teacher team 
meetings 

Meetings where interdisciplinary core 
teachers discuss shared students 

0: Do not/rarely occur 

0.2: Occur monthly 

0.4: Occur biweekly 

0.6: Occur weekly 

0.8: Occur multiple times a week 

1: Occur daily 

0.8: Adequate 

0.8: Occur multiple 
times a week 

Site-based team 
standards 

Site-based team (administrator, school 
transformation facilitator, project 
manager, and site coordinator) 
standards for collaboration, 
communication, and decision making 

0: Not in place 

0.5: Partially/in process 

1: In place 

0.5: Adequate 

0.5: Parti ally/in 
progress 

Site-based team 
meetings 

Brief meetings for site-based team to 
review program implementation 
(approximately 30 minutes) 

0: Once a month or less 
0.5: Biweekly 

1: Weekly or more frequently 

0.5: Adequate 

0: Once a month 
or less 

4x4 block 
(high school only) 

4 class periods of 75 to 90 minutes that 
meet daily (or at least 4 days a week) 

0: No 

0.5: Flybrid/acceptable 

alternative 

1: Yes 

0.5: Adequate 

0: No 


(continued) 







Appendix Table A.2 (continued) 

SOURCE: Diplomas Now fidelity of implementation matrix 2013. 

NOTES: This table only provides Strong Learning Environments components that apply to 
high schools. One middle school-specific component of Strong Learning Environments has 
been omitted. 

See Appendix Table B.2 in the first report of this evaluation (Corrin et al., 2014a) for 
fidelity scales and criteria for adequacy for all components in the fidelity of implementation 
matrix. 

“Scales have been standardized to assist with the calculation of implementation metrics. 


Appendix Table A.2. First, the components are standardized to a 0-to-l scale (for example, a 
component scaled from 0 to 2 is recoded so that 0 = 0, 1 = 0.5, and 2 = 1), then averaged 
together. In this example, there are five components of the Strong Learning Environments input: 
small learning communities, interdisciplinary teams, Diplomas Now site-based meetings, 
Diplomas Now site-based collaboration, and 4x4 block scheduling. The average of the fidelity 
scale responses for these components equals 0.46 (that is, 1 + 0.8 + 0 + 0.5 + 0, divided by 5). 
This is the school’s fidelity score for Strong Learning Environments. Scores for the remaining 
inputs are calculated similarly, and the average of all input scores provides the school’s fidelity 
score. The average of fidelity scores from all schools provides an overall score that estimates the 
proportion of the Diplomas Now model (from 0 to 1) implemented across all DN schools during 
the first year. 


Component-Level Fidelity Matrix Findings 

Appendix Table A.3 provides component-level findings (the percentage of schools that ade- 
quately implemented each component) for the 1 1 1 components in the model, for Year 1 and 
Year 2, organized by pillar and input. It also denotes which components were identified by the 
Diplomas Now Implementation Support Team as critical to the early implementation of the 
program. 
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Diplomas Now 
Appendix Table A.3 


Component-Level Fidelity of Implementation at Diplomas Now Schools, Year 1 and Year 2 


Component Description" 1 

Identified _ 
as Critical? 

Percentage of 
Schools Adequately 
Implementing Component 
Year 1 Year 2 Difference 

Pillar I. Teacher Teams and Small Learning Communities: Strong Learning Environments 

Site-based team (administrator, school transformation facilitator, project manager, and site 
coordinator) had standards for collaboration, communication, and decision making 

No 

96.9 

100.0 

3.1 

Meetings for site-based team to review program implementation of approx. 30 minutes occurred at 
least biweekly 

Yes 

78.1 

71.0 

-7.2 

High schools had 4 class periods of 75 to 90 minutes that met daily (or at least 4 days a week) or 
acceptable hybrid 

Yes 

86.7 

73.3 

-13.3 

Middle schools provided extended class periods of 70 to 90 minutes for core academic classes 

Yes 

52.9 

68.8 

15.8 

Interdisciplinary teams of teachers were organized into small learning communities and worked 
with the same small group of students 

Yes 

68.8 

77.4 

8.7 

Meetings where interdisciplinary core teachers discussed shared students occurred multiple 
times per week 

Yes 

25.0 

29.0 

4.0 


Pillar II. Curriculum and Instruction with Professional Development: Professional Development and Peer Coaching 


High schools provided professional development and job -embedded support for 


freshman seminar teachers 

No 

73.3 

53.3 

-20.0 

Math coaches provided at least one period of support per week to each mathematics teacher 

Yes 

59.4 

51.6 

-7.8 

Instructional coaching cycle (plan, coteach, model, and debrief) completed by math coach 
with teachers, on average, at least biweekly 

No 

43.8 

38.7 

-5.0 

English/language arts coaches provided at least one period of support per week to each 
English/language arts teacher 

Yes 

62.5 

61.3 

-1.2 
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Appendix Table A.3 (continued) 


Percentage of 
Schools Adequately 
Identified Implementing Component 

Component Description 1 as Critical? Year 1 Year 2 Difference 

Pillar II. Curriculum and Instruction with Professional Development: Professional Development and Peer Coaching 


Instructional coaching cycle (plan, coteach, model, and debrief) completed by English/language 
arts coach with teachers, on average, at least biweekly 

No 

43.8 

45.2 

1.4 

Pillar II. Curriculum and Instruction with Professional Development: Curriculum for College Readiness 

High schools offered a separate academy for 9th-graders with its own administrators, teachers, 
and counselors 

Yes 

60.0 

60.0 

0.0 

Middle schools offered the reform program on school climate to 6th-grade students 

No 

41.2 

37.5 

-3.7 

Middle schools offered the reform curriculum on school success skills, Mastering the Middle Grades, 
to at least all 6th-grade students 

No 

11.8 

56.3 

44.5 

High schools offered a freshman seminar to 9th-graders during their first semester 

Yes 

73.3 

53.3 

-20.0 

High schools offered at least 6 units of the freshman seminar to 9th-grade students 

No 

46.7 

26.7 

-20.0 

Middle schools met high standards of implementation of Common Core Standards in mathematics 

No 

76.5 

81.3 

4.8 

Middle schools implemented an evidence-based mathematics curriculum or adequate alternative 

No 

58.8 

56.3 

-2.6 

High schools implemented the Transition to Advanced Mathematics reform curriculum for 
9th-grade students 

No 

60.0 

46.7 

-13.3 

High schools offered Transition to Advance Mathematics in classes that met at least 4 times per 
week for at least 70 minutes per meeting 

No 

46.7 

40.0 

-6.7 

High schools implemented Geometry Foundations reform curriculum for 1 Oth-grade students 

No 

13.3 

33.3 

20.0 
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Appendix Table A.3 (continued) 


Percentage of 
Schools Adequately 
Identified Implementing Component 


Component Description 3 

as Critical? 

Year 1 

Year 2 

Difference 

Pillar II. Curriculum and Instruction with Professional Development: Curriculum for College Readiness 

High schools offered Geometry Foundations in classes that met at least 4 times per week for 
at least 70 minutes per meeting 

No 

13.3 

13.3 

0.0 

High schools implemented Algebra II Foundations reform curriculum for 1 lth-grade students 

No 

0.0 

6.7 

6.7 

High schools offered Algebra II Foundations in classes that met at least 4 times per week for 
at least 70 minutes per meeting 

No 

0.0 

0.0 

0.0 

Middle schools offered the Computer and Team Assisted Mathematics Acceleration (CATAMA) 
curriculum that provides additional math instruction and support to underprepared students 

No 

29.4 

18.8 

-10.7 

High schools offered the Computer and Team Assisted Mathematics Acceleration (CATAMA) 
program that provides additional math instruction and support to underprepared students 

No 

20.0 

26.7 

6.7 

Middle schools offered the Student Team Literature reform curriculum or adequate alternative 

Yes 

41.2 

50.0 

8.8 

Middle schools offered the Savvy Readers' Lab reform curriculum or adequate alternative 

Yes 

41.2 

25.0 

-16.2 

High schools offered the Strategic Reading reform curriculum for 9th-grade students 

No 

60.0 

53.3 

-6.7 

High schools offered Strategic Reading in classes that met at least 4 times per week for 
at least 70 minutes per meeting 

No 

46.7 

33.3 

-13.3 

High schools implemented the Reading and Writing in Your Career reform curriculum 
for lOth-grade students 

No 

0.0 

20.0 

20.0 
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Appendix Table A.3 (continued) 


Percentage of 
Schools Adequately 
Identified Implementing Component 


Component Description" 1 

as Critical? 

Year 1 

Year 2 

Difference 

Pillar II. Curriculum and Instruction with Professional Development: Curriculum for College Readiness 

High schools offered Reading and Writing in Your Career in classes that met at least 4 times 
per week for at least 70 minutes per meeting 

No 

0.0 

13.3 

13.3 

High schools offered the College Prep Reading and Writing reform curriculum 
for 1 lth-grade students 

No 

0.0 

0.0 

0.0 

High schools offered College Prep Reading and Writing in classes that met at least 4 times per 
week for at least 70 minutes per meeting 

No 

0.0 

0.0 

0.0 

High schools offered the Accelerating Literacy for Adolescents Lab or adequate alternative to 
9th-grade students who were significantly below grade level 

No 

13.3 

20.0 

6.7 

Pillar III. Tiered Student Supports: Tiered Intervention Model 

A coordinated Early Warning Indicator (EWI) data system was in place that tracked student 
attendance, behavior, and course performance and alerted teachers as students began to fall 
off the graduation track 

Yes 

93.8 

90.3 

-3.4 

Interdisciplinary EWI team meetings to discuss students demonstrating off-track indicators 
occurred at least biweekly 

Yes 

90.6 

80.6 

-10.0 

A plan for integrating the use of the EWI system and scheduling EWI meetings was in place 

No 

75.0 

83.9 

8.9 

Pillar III. Tiered Student Supports: Student Supports 

Additional in-class support from City Year AmeriCorps members was offered in mathematics 
classrooms year-round 

Yes 

100.0 

100.0 

0.0 

At least 75% of City Year AmeriCorps members were embedded in mathematic classrooms 

Yes 

62.5 

61.3 

-1.2 

City Year AmeriCorps members were in mathematic classrooms at least three times per week 

Yes 

100.0 

100.0 

0.0 
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Appendix Table A.3 (continued) 


Percentage of 
Schools Adequately 
Identified Implementing Component 


Component Description 1 

as Critical? 

Year 1 

Year 2 

Difference 

Pillar III. Tiered Student Supports: Student Supports 





At least 75% of math classrooms had City Year AmeriCorps members embedded 

Additional in-class support from City Year AmeriCorps members was offered in English/language arts 

Yes 

40.6 

41.9 

1.3 

classrooms year-round 

At least 75% of City Year AmeriCorps members were embedded in English/language 

Yes 

100.0 

100.0 

0.0 

arts classrooms 

Yes 

65.6 

64.5 

-1.1 

City Year AmeriCorps members were in English/language arts classrooms at least three times per week 

Yes 

100.0 

100.0 

0.0 

At least 75% of English/language arts classrooms had City Year AmeriCorps members embedded 

Yes 

50.0 

41.9 

-8.1 

Mathematics tutoring structure and schedule was in place 

Yes 

71.9 

71.0 

-0.9 

City Year AmeriCorps members were integrated into mathematics tutoring structure 

Ratio of students receiving mathematics tutoring to City Year AmeriCorps members was equal to or 

Yes 

62.5 

64.5 

2.0 

less than 7 to 1 

Yes 

50.0 

48.4 

-1.6 

Literacy tutoring structure and schedule was in place 

Yes 

75.0 

71.0 

-4.0 

City Year AmeriCorps members were integrated into literacy tutoring structure 

Ratio of students receiving literacy tutoring to City Year AmeriCorps members was equal to or 

Yes 

59.4 

61.3 

1.9 

less than 7 to 1 

Yes 

59.4 

54.8 

-4.5 

City Year AmeriCorps members were integrated into the school's attendance program 

Ratio of students receiving attendance coaching to City Year AmeriCorps members was equal to or 

Yes 

71.9 

74.2 

2.3 

less than 1 to 1 

Yes 

93.8 

100.0 

6.3 


(continued) 
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Component Description 11 

Identified _ 
as Critical? 

Percentage of 
Schools Adequately 
Implementing Component 
Year 1 Year 2 Difference 

Pillar III. Tiered Student Supports: Student Supports 

Ratio of students receiving behavior coaching to City Year AmeriCorps members was equal to or 
less than 1 to 1 

Yes 

93.8 

87.1 

-6.7 

After-school program or extended learning time was offered to a subset of the school's students 

Yes 

96.9 

96.8 

-0.1 

City Year AmeriCorps members consistently recmited students not on track to graduate to attend 
after-school programs 

No 

78.1 

77.4 

-0.7 

At least 25% of students regularly attended after-school programs 

No 

56.3 

61.3 

5.0 

After-school programs were implemented all year 

No 

46.9 

51.6 

4.7 

School integrated City Year AmeriCorps members into existing after-school programs 

No 

40.6 

45.2 

4.5 

Existing after-school programs with integrated City Year AmeriCorps members were implemented 
all year 

No 

43.8 

45.2 

1.4 

At least four whole-school activities (for example, health fairs or career days) were conducted during 
the school year 

Yes 

100.0 

100.0 

0.0 

Pillar III. Tiered Student Supports: Student Case Management 

At least 75% of students identified through the EWI system as needing Tier III support received 
case management 

Yes 

71.9 

77.4 

5.5 

At least 75% of case-managed students were provided with individual student plans 

Yes 

90.6 

96.8 

6.2 

At least 75% of case-managed students identified with attendance issues were 
provided with attendance interventions 

Yes 

75.0 

67.7 

-7.3 

At least 75% of case-managed students identified with behavior problems were 
provided with behavior interventions 

Yes 

90.6 

74.2 

-16.4 


(continued) 
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Component Description 1 

Identified _ 
as Critical? 

Percentage of 
Schools Adequately 
Implementing Component 
Year 1 Year 2 Difference 

Pillar III. Tiered Student Supports: Student Case Management 

At least 75% of case-managed students identified with academic needs were 
provided with academic interventions 

Yes 

65.6 

71.0 

5.4 

At least 75% of case-managed students identified with basic needs (for example, for food, 
clothing, or shelter) were provided with basic needs/resource interventions 

No 

87.5 

74.2 

-13.3 

At least 5% of students in schools with more than 1,000 students and 10% of students in schools 
with fewer than 1,000 students received case management 

No 

65.6 

74.2 

8.6 

Communities In Schools site coordinators provided whole-school enrichment/motivation services 

No 

96.9 

96.8 

-0.1 

Communities In Schools site coordinators provided whole-school family engagement and 
strengthening services 

No 

84.4 

71.0 

-13.4 

Communities In Schools site coordinators provided whole-school life skills and social 
development services 

No 

75.0 

51.6 

-23.4 

Communities In Schools site coordinators provided whole-school college and career services 

No 

68.8 

67.7 

-1.1 

Communities In Schools site coordinators provided whole-school professional physical 
health services 

No 

46.9 

38.7 

-8.2 

Communities In Schools site coordinators provided whole-school community-service opportunities 

No 

37.5 

25.8 

-11.7 

Communities In Schools site coordinators provided whole-school professional mental health services 

No 

37.5 

9.7 

-27.8 

Pillar IV. Can-Do Culture and Climate: Integrated On-Site Support 

Mathematics instructional facilitator provided technical assistance 

Yes 

100.0 

100.0 

0.0 

English/language arts instructional facilitator provided technical assistance 

Yes 

100.0 

100.0 

0.0 

School and student support services (S4) facilitator provided technical assistance 

Yes 

100.0 

100.0 

0.0 
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Component Description" 1 

Identified _ 
as Critical? 

Percentage of 
Schools Adequately 
Implementing Component 
Year 1 Year 2 Difference 

Pillar IV. Can-Do Culture and Climate: Integrated On-Site Support 





School transformation facilitator began serving school prior to the first day of school 

No 

81.3 

87.1 

5.8 

Instructional mathematics coach worked at the school at least half time 

Yes 

75.0 

71.0 

-4.0 

Instructional English/language arts coach worked at the school least half time 

Yes 

81.3 

80.6 

-0.6 

Communities In Schools site coordinator was in place 

Yes 

100.0 

100.0 

0.0 

Site coordinator began serving the school before the first school progress report 

Yes 

81.3 

93.5 

12.3 

At least 75% of City Year AmeriCorps members were retained throughout the school year 

Yes 

100.0 

93.5 

-6.5 

Ratio of students to City Year AmeriCorps members was less than 50 to 1 

Yes 

78.1 

90.3 

12.2 

Ratio of second-year City Year AmeriCorps members to first-year City Year AmeriCorps members was 
equal to or greater than 1 to 1 0 

No 

78.1 

83.9 

5.7 


Pillar IV. Can-Do Culture and Climate: Family and Community Involvement 


School offered workshops to parents at least once during the school year 

Teachers sent information to parents on how to help their children at least once during 

No 

87.5 

90.3 

2.8 

the school year 

No 

78.1 

83.9 

5.7 

School scheduled parent-teacher conferences (with all core teachers) for each student's family 
School had an Action Team for Partnerships, or its equivalent, to develop, implement, and evaluate 

No 

75.0 

71.0 

-4.0 

work on family and community engagement 

No 

56.3 

54.8 

-1.4 

School formally recruited parents and trained them to work as school volunteers 

No 

18.8 

41.9 

23.2 

Actual parent engagement practices somewhat matched ideal practices 

Yes 

65.6 

67.7 

2.1 
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Component Description 1 

Identified _ 
as Critical? 

Percentage of 
Schools Adequately 
Implementing Component 
Year 1 Year 2 Difference 

Program Staff Training and Development 





Talent Development staff members had access to Communities In Schools staff members 

Yes 

100.0 

96.8 

-3.2 

Talent Development staff members had access to City Year staff members 

Yes 

100.0 

96.8 

-3.2 

Communities In Schools staff members had access to Talent Development staff members 

Yes 

100.0 

100.0 

0.0 

Communities In Schools staff members had access to City Year staff members 

Yes 

100.0 

100.0 

0.0 

City Year staff members had access to Talent Development staff members 

Yes 

93.8 

96.8 

3.0 

City Year staff members had access to Communities In Schools staff members 

Yes 

93.8 

96.8 

3.0 

Professional development plan for on-site training for teachers and school-based staff members 
focused on curriculum, school climate, and teaming was at least partially implemented 

Yes 

81.3 

83.9 

2.6 

Professional development plan for school-based team to engage in continuing professional 
development opportunities throughout the school year was at least partially implemented 

Yes 

78.1 

83.9 

5.7 

School-based team attended a three-day summer institute or alternate training session aimed at 
helping a new Diplomas Now school start strong 

Yes 

78.1 

93.5 

15.4 

Joint planning sessions for school administrators and teachers occurred prior to the start 
of the school year 

Yes 

59.4 

93.5 

34.2 

City Year AmeriCorps members participated in relevant teacher professional development 
opportunities at least once or twice during school year 

Yes 

93.8 

96.8 

3.0 

School transformation facilitator attended five-day summer training session or alternate training 

Yes 

96.9 

100.0 

3.1 

School's mathematics coach attended three-day summer training session or alternate training 

No 

43.8 

58.1 

14.3 
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Percentage of 
Schools Adequately 
Identified Implementing Component 


Component Description" 1 

as Critical? 

Year 1 

Year 2 

Difference 

Program Staff Training and Development 

School's English/language arts coach attended three-day summer training session or 
alternate training 

No 

46.9 

64.5 

17.6 

Site coordinators completed an 1 1 -module online course, approximately 1.5-2 hours per module 

Yes 

40.6 

67.7 

27.1 

City Year AmeriCorps members received training in the use of data to identify interventions 

Yes 

56.3 

77.4 

21.2 

City Year AmeriCorps members received ongoing support in the use of data to identify interventions 

Yes 

53.1 

74.2 

21.1 

City Year AmeriCorps members participated in EWI -related training sessions at least three 
times per year 

No 

34.4 

29.0 

-5.3 


SOURCES: Diplomas Now fidelity of implementation program staff surveys and records, 2012, 2013, and 2014. Program staff surveys and records 
include: the Diplomas Now Implementation Support Team survey, Talent Development school transformation facilitator survey, City Year 
program manager survey, Communities In Schools site coordinator survey, City Year records, and Community In Schools records. 

NOTE: Average percentage of sites adequate for middle and high school-specific components reflect only the appropriate number of middle (17) 
or high (15) schools. One DN school closed prior to the second year of implementation and is not included in the Year 2 analyses. 

a Embedded within each component description is the criterion for adequate implementation as defined by the Implementation Support Team. 

See Appendix Table B.2 in the first report of this evaluation (Corrin et ah, 2014a) for fidelity scales and criteria for adequacy for all components in 
the fidelity of implementation matrix. 



Appendix B 

Service Contrast Supplemental Materials 




This appendix includes supplementary materials connected to the service contrast analyses 
found in Chapters 2 and 3. First, it discusses the teacher and administrator survey data used for 
these analyses and the response rates associated with these data sources. The second section 
describes the creation of measures used in the service contrast analyses where more than one 
survey item was combined. The following section discusses the methodology utilized in the 
service contrast analyses. Finally, supplemental service contrast tables are shared. 


Data Sources and Survey Response Rates 

The measures used in the service contrast analyses come from teacher and administrator 
(principal and assistant principal) surveys administered at all study schools. Teacher and 
administrator surveys used in analyses of Year 2 implementation were administered online or in 
pencil-and-paper format during the spring of the second year of implementation (2013 for 
“wave 1” schools — schools that joined the study in the 201 1-2012 school year — and 2014 for 
“wave 2” schools — schools that joined the study in the 2012-2013 school year). Teacher 
surveys used in the analyses of Year 1 implementation were administered in the spring of the 
first year of implementation (2012 for wave 1 schools and 2013 for wave 2 schools). Copies of 
both teacher and administrator surveys can be found in the Data Collection Instrument Supple- 
ment to the first report. 1 

Appendix Table B.l describes the response rate and sample size for the teacher and 
administrator surveys. The first set of columns shows the school-level response rates for study 
schools assigned to implement the Diplomas Now model (DN schools), study schools not 
assigned to implement it (non-DN schools), and all schools (DN schools and non-DN schools 
combined). School-level response rates were calculated by dividing the number of schools with 
at least one survey respondent by the total number of schools. The second set of columns 
demonstrates the individual-level response rates within those same groups of schools. Tables 
2.3, 2.5, 2.7, and 2.9 use data from the Year 2 teacher survey. Appendix Tables B.2-B.5 use 
data from the Year 2 administrator survey. Data from the Year 1 teacher survey are in Appendix 
Tables B.6-B.9, and Appendix Tables B.10-B.13 use data from sixth- and ninth-grade teachers 
during the second year of implementation. (Tables are at the end of this appendix.) 

During the first year of implementation, all schools were included in the teacher survey 
analyses since each school had at least one survey respondent. In the second year of implemen- 
tation, one DN school closed and is not included in any of the service contrast analyses. One 
school was dropped from the Year 2 teacher survey analyses for sixth and ninth grades because 
none of the teachers in those grade levels responded to the survey. 


'Corrin et al. (2014b), available at www.mdrc.org. 
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Diplomas Now 
Appendix Table B.l 

Teacher and Administrator Survey Response Rates, 
School and Individual 



School Level 


Individual Level 



DN 

Non-DN 

All 

DN 

Non-DN 

All 


Schools 

Schools Schools 

Schools 

Schools 

Schools 

Panel A: Teachers 







Y ear 2 response rate (%) 

96.9 

100.0 

98.4 

75.6 

78.8 

77.3 

Sample size 

31 

30 

61 

1,476 

1,652 

3,128 

Y ear 1 response rate (%) 

100.0 

100.0 

100.0 

80.8 

85.1 

83.0 

Sample size 

32 

30 

62 

1,635 

1,732 

3,367 

Panel B: Administrators 







Y ear 2 response rate (%) 

96.9 

100.0 

98.4 

84.5 

83.2 

83.9 

Sample size 

31 

30 

61 

110 

107 

217 


SOURCE: Survey administration documentation, 2012, 2013, and 2014. 

NOTES: One DN school closed prior to the second year of implementation and is not included in the 
Year 2 analyses. First-year administrator response rates are not included in this table because these data 
are not referred to in this report. 


The construction of the analytic samples from the pool of eligible teachers and adminis- 
trators is presented in Appendix Figures B.l through B.3. Appendix Figure B.l shows that 
3,128 eligible teachers were identified during the second year of implementation (on average, 
51 teachers per school). Of these eligible teachers, 2,418 (77 percent) responded to the survey 
questionnaire. Among the survey respondents, 2,288 were teachers who were kept in the 
analysis, while 130 were nonteachers (for example, counselors or academic advisers) who were 
dropped from the analysis. The primary analysis sample in Year 2 is composed of the 1,818 
teachers who taught sixth, seventh, ninth, or tenth grade, since these grade levels were the most 
fully served by Diplomas Now during the second year of implementation. In addition, for 
certain school practices Diplomas Now only targets specific academic subjects, which results in 
three subsamples of teachers. Within the primary analytic sample, there were 1,269 core subject 
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teachers, 827 math and English/language arts teachers, and 465 high school math and Eng- 
lish/language arts teachers. 2 For the secondary analysis on the transition years in Year 2, the 
study team examined 1,339 sixth- and ninth-grade teachers, of whom 853 taught core subjects, 
577 taught math and English/language arts, and 332 taught high school math and Eng- 
lish/language arts. 

As shown in Appendix Figure B.2, in the first year of implementation the study team 
identified 3,367 eligible teachers from the 62 participating schools (on average, 54 teachers per 
school). Out of the eligible pool of teachers, there were 2,795 survey respondents (an 83 percent 
response rate). The majority of survey respondents (2,643) were teachers who remained in the 
analysis, but 152 nonteacher respondents were excluded from the analysis. Diplomas Now 
focused its attention on students in transition years (sixth and ninth grades) during the first year 
of implementation, so the primary analysis sample in Year 1 consists of the 1,620 teachers 
instructing those grade levels. Among the primary sample of sixth- and ninth-grade teachers in 
Year 1, there were 1,043 core subject teachers, 685 math and English/language arts teachers, 
and 404 high school math and English/language arts teachers. 

Appendix Figure B.3 illustrates that 217 eligible administrators (principals and assistant 
principals) were identified during the second year of implementation. Of these eligible adminis- 
trators, 182 (84 percent) responded to the survey questionnaire. This group was then divided 
into the 170 administrator respondents who were kept in the analysis and the 12 nonadministra- 
tor respondents who were dropped from the analysis. Because a few aspects of the Diplomas 
Now model only applied to high schools, a subsample of 102 high school administrators was 
also examined for some service contrast measures. 


Service Contrast Measure Creation and Factor Analysis 

Many of the tables and graphs in Chapters 2 and 3 as well as the tables in this appendix com- 
pare the programs and services in DN and non-DN schools during the first two years of imple- 
mentation. All of these exhibits are based on surveys of teachers and administrators. In most 
cases, single survey items were used in the analyses, but in some cases, multiple survey items 
were combined to create a single measure. The following discussion describes the creation of 
measures that were constructed out of more than one survey item and the factor analysis for 
those measures. 


2 “Core” academic areas are identified as math. English/language arts, sciences, and social studies. 
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Analysis Analysis Teacher Respondent Eligibility 

Subsamples Sample Sample Sample Pool 


Diplomas Now 


Appendix Figure B.l 

Construction of the Sample of Teachers for the Service Contrast Analyses 

in Year 2 
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Analysis Analysis Teacher Respondent Eligibility 

Subsamples Sample Sample Sample Pool 


Diplomas Now 


Appendix Figure B.2 

Construction of the Sample of Teachers for the Service Contrast Analyses 

in Year 1 
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Diplomas Now 


Appendix Figure B.3 

Construction of the Sample of Administrators for the Service Contrast Analyses in 

Year 2 



Expected number of administrator 
survey respondents 

(n = 217) 


Nonadministrator 
respondents 
dropped from 
sample 

(n = 12) 



Number of survey respondents 

(n = 182) 


Teacher Survey Items 

Table 2.3, Appendix Table B.6, and Appendix Table B.10 include the teacher survey 
measure: “Core teachers reported hours spent each week collaborating with an interdisciplinary 
team and teaching coordinated content across disciplines.” The value for the construct was 
calculated by taking an average of the survey item responses and then rescaling to a 0-10 scale. 
Responses that stated “Not occurring in your school [this year]” were coded as 0. (Three items, 
Cronbach’s alpha = 0.88.) 
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During the [current] school year, how many hours did you spend each week 
participating in the following activities at your school? 

_ Common planning with an interdisciplinary team of teachers that shared 
the same students. 

- Collaborating with an interdisciplinary team of teachers to determine ap- 
proaches to respond to student needs. 

- Teaching academic content coordinated across an interdisciplinary team, 
academy or small learning community. 

(Scale: 0 = Not occurring in your school [this year], 1 = None, 2 = Less than 1 
hour a week, 3 = 1-2 hours each week, 4 = 2-3 hours each week, 5 = More than 
3 hours each week) 

Table 2.5, Appendix Table B.7, and Appendix Table B.ll include the teacher survey 
measure: “Average times per month, math and English/language arts teachers reported working 
with an instructional coach.” The value for the construct was calculated by weighting the 
frequency per month and then summing the weighted survey items in order to get a total times- 
per-month measure. For example, 0 = never, 0.5 = more than once a year but not monthly, 1 = 
at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 12 = more 
than once a week but not daily, 20 = daily. Responses that stated “not occurring in your school 
[this year]” were coded as 0 (two items). 

During the [current] school year, how often did the following occur at your school? 

_ A Math coach provided you with instructional mentoring and support. 

_ A Language Arts coach provided you with instructional mentoring and 
support. 

(Scale: 0 = Not occurring in your school [this year], 1 = Never, 2 = More than 
once a year, but not monthly, 3 = At least once a month, 4 = More than once a 
month, but not weekly, 5 = Once a week, 6 = More than once a week, but not 
daily, 7 = Daily) 


Table 2.5, Appendix Table B.7, and Appendix Table B.ll include the teacher survey 
measure: “Average times per month, math and English/language arts teachers reported receiv- 
ing support from a school leader or a coach.” The value for the construct was calculated by 
weighting the frequency per month and then summing the weighted survey items in order to get 
a total times-per-month measure. For example, 0 = never, 0.5 = more than once a year but not 
monthly, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a 
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week, 12 = more than once a week but not daily, 20 = daily. Responses that stated “not occur- 
ring in your school [this year]” were coded as 0 (three items). 

During the [current] school year, how often did the following occur at your school? 

_ School leaders (including peer coaches, mentors, and/or facilitators) pro- 
vided you with instructional mentoring and support. 

_ A Math coach provided you with instructional mentoring and support. 

_ A Language Arts coach provided you with instructional mentoring and 
support. 

(Scale: Not occurring in your school [this year], 1 = Never, 2 = More than once 
a year, but not monthly, 3 = At least once a month, 4 = More than once a 
month, but not weekly, 5 = Once a week, 6 = More than once a week, but not 
daily, 7 = Daily) 


Table 2.5, Appendix Table B.7, and Appendix Table B.ll include the teacher survey 
measure: “Math and English/language arts teachers reported struggling students in their courses 
received additional classes to catch up with their peers.” The value for the construct was 
calculated by combining the two survey questions based on the subject that a teacher instructs. 
The first question in the construct below was used for math teachers, while the second question 
in the construct below was used for English/language arts teachers. Responses that stated “don’t 
know/does not apply” were coded as missing values (two items). 

Please indicate “yes” or “no” to the following statements describing your school or 

your teaching during the [current] school year. 

_ Students who were struggling in math received a “double dose” of math 
or a supplemental math course to help them catch up to their classmates. 

_ Students who were struggling in reading received a “double dose” of 
language arts or a supplemental literacy course to help them catch up to 
their classmates. 

(Scale: 0 = No, 1 = Yes, 3 = Don’t know/Does not apply) 

Table 2.5, Appendix Table B.7, and Appendix Table B.ll include the teacher survey 
measure: “Math and English/language arts teachers reported the average frequency they used a 
variety of student-centered teaching strategies.” The value for the construct was calculated by 
taking an average of the survey item responses and then rescaling to a 0-10 scale. (Four items, 
Cronbach’s alpha = 0.71.) 
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During the past month, how often did the following instructional activities occur in 
most of your classes? 

_ Students worked in small groups or pairs. 

_ Students worked on individual or group projects that lasted several days. 

_ Students were engaged in different learning activities at the same time. 

_ You provided demonstrations, modeling of strategies, and mini-lessons. 

(Scale: 1 = Never, 2 = At least once a month, 3 = More than once a month, but 
not weekly, 4 = Once a week, 5 = More than once a week, but not daily, 6 = 
Daily) 


Table 2.5, Appendix Table B.7, and Appendix Table B.ll include the teacher survey 
measure: “Math and English/language arts teachers reported the average frequency students 
applied classroom activities to real-life issues and used critical thinking and reasoning skills.” 
The value for the construct was calculated by taking an average of the survey item responses 
and then rescaling to a 0-10 scale. (Three items, Cronbach’s alpha = 0.72.) 

During the past month, how often did the following instructional activities occur in 
most of your classes? 

_ Students used manipulative materials to illustrate concepts. 

_ Students applied classroom activities to real-life issues. 

_ Tasks required students to use critical thinking and reasoning skills to 
solve problems. 

(Scale: 1 = Never, 2 = At least once a month, 3 = More than once a month, but 
not weekly, 4 = Once a week, 5 = More than once a week, but not daily, 6 = 

Daily) 


Table 2.5, Appendix Table B.7, and Appendix Table B.ll include the teacher survey 
measure: “Math and English/language arts teachers reported the average frequency academic 
courses included career applications and exploration.” The value for the construct was calculat- 
ed by taking an average of the survey item responses and then rescaling to a 0-10 scale. (Two 
items, Cronbach’s alpha = 0.89.) 

During the past month, how often did the following instructional activities occur in 
most of your classes? 

_ You taught academic content that blended career applications across ac- 
ademic courses. 
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_ You taught a curriculum that included career exploration and planning. 

(Scale: 1 = Never, 2 = At least once a month, 3 = More than once a month, but 
not weekly, 4 = Once a week, 5 = More than once a week, but not daily, 6 = 

Daily) 

Table 2.7, Appendix Table B.8, and Appendix Table B.12 include the teacher survey 
measure: “Core teachers reported the average frequency they used attendance, behavior, and 
course performance data to target at-risk students.” The value for the construct was calculated 
by taking an average of the survey item responses and then rescaling to a 0-10 scale. (Three 
items, Cronbach’s alpha = 0.90.) 

During the past month, how often did you analyze the following types of data to 
target at-risk students? 

_ Course performance data. 

_ Student attendance data. 

_ Student behavior data. 

(Scale: 1 = Never, 2 = At least once a month, 3 = More than once a month, but 
not weekly, 4 = Once a week, 5 = More than once a week, but not daily, 6 = 

Daily) 

Table 2.7, Appendix Table B.8, and Appendix Table B.12 include the teacher survey 
measure: “Average times per month, math and English/language arts teachers reported students 
received academic help in class from volunteers.” The value for the construct was calculated by 
weighting the frequency per month and then summing the weighted survey items in order to get 
a total times-per-month measure. For example, 0 = never, 1 = at least once a month, 2 = more 
than once a month but not weekly, 5 = once a week, 12 = more than once a week but not daily, 
20 = daily (five items). 

During the past month, how often did each of the following volunteer activities 
occur with students in your classes? 

_ Literacy one-on-one tutoring. 

_ Math one-on-one tutoring. 

_ Literacy small group tutoring. 

_ Math small group tutoring. 

_ Whole class academic support. 
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(Scale: 1 = Never, 2 = At least once a month, 3 = More than once a month, but 
not weekly, 4 = Once a week, 5 = More than once a week, but not daily, 6 = 

Daily) 

Table 2.7, Appendix Table B.8, and Appendix Table B.12 include the teacher survey 
measure: “Core teachers reported the frequency academic and non-academic services were 
coordinated for students at-risk of dropping out of school.” The value for the construct was 
calculated by taking an average of the survey item responses and then rescaling to a 0-10 scale. 
(Two items, Cronbach’s alpha = 0.95.) 

To the best of your ability, please indicate how frequently the following activities 
occurred at your school during the [current] school year. 

_ Academic and non-academic services for students at-risk of dropping out 
of school were coordinated to meet students’ needs. 

_ Academic and non-academic services for students at-risk of dropping out 
of school were monitored to ensure students’ needs were met. 

(Scale: 1 = Never, 2 = Rarely, 3 = Sometimes, 4 = Often, 5 = Always, as need- 
ed) 

Table 2.9, Appendix Table B.9, and Appendix Table B.13 include the teacher survey 
measure: “Average times per month, math and English/language arts teachers reported City 
Year AmeriCorps members, college students, or volunteers from organized programs worked 
with students.” The value for the construct was calculated by weighting the frequency per 
month and then summing the weighted survey items in order to get a total times-per-month 
measure. For example, 0 = never, 1 = at least once a month, 2 = more than once a month but not 
weekly, 5 = once a week, 12 = more than once a week but not daily, 20 = daily (three items). 

During the past month, how often did each of the following volunteers work with 
students in at least some of your classes? 

_ Students from local college/teacher preparation programs. 

_ City Year corps members. 

_ Members of organized volunteer programs or other City Year Ameri- 
Corps members. 

(Scale: 1 = Never, 2 = At least once a month, 3 = More than once a month, but 
not weekly, 4 = Once a week, 5 = More than once a week, but not daily, 6 = 

Daily) 
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Table 2.9, Appendix Table B.9, and Appendix Table B.13 include the teacher survey 
measure: “Average times per month, math and English/language arts teachers reported any 
volunteer worked with students.” The value for the construct was calculated by weighting the 
frequency per month and then summing the weighted survey items in order to get a total times- 
per-month measure. For example, 0 = never, 1 = at least once a month, 2 = more than once a 
month but not weekly, 5 = once a week, 12 = more than once a week but not daily, 20 = daily 
(eight items). 

During the past month, how often did each of the following volunteers work with 
students in at least some of your classes? 

_ Parents/guardians. 

_ Community members. 

_ Students from local college/teacher preparation programs. 

_ City Year corps members. 

_ Members of organized volunteer programs or other City Year Ameri- 
Corps members. 

_ Peer tutors (students from other classes in the school). 

_ Other community members. 

_ Others, please specify. 

(Scale: 1 = Never, 2 = At least once a month, 3 = More than once a month, but 
not weekly, 4 = Once a week, 5 = More than once a week, but not daily, 6 = 

Daily) 

Administrator Survey Items 

Appendix Table B.3 includes the administrator survey measure: “Average frequency of 
opportunities to participate in a principal/leader support network or other professional develop- 
ment.” The value for the construct was calculated by taking an average of the survey item 
responses and then rescaling to a 0-10 scale. (Two items, Cronbach’s alpha = 0.79.) 

During the [current] school year, how often did principals and assistant principals at 
your school have the opportunity to participate in the following: 

_ Professional development activities. 

_ Principal/leader support network. 

(Scale: 1 = Never or once a year, 2 = More than once a year, but not monthly, 3 
= At least monthly, 4 = More than once a month, but not weekly, 5 = At least 
once a week, 6 = More than once a week, 7 = Daily) 
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Appendix Table B.3 includes the administrator survey measure: “Average frequency 
students were provided with information about career fairs, job shadowing, internship opportu- 
nities, or college readiness.” The value for the construct was calculated by taking an average of 
the survey item responses and then rescaling to a 0-10 scale. (Two items, Cronbach’s alpha = 
0.87.) 


During the [current] school year, how frequently were the following supports 
provided to students at your school? 

_ Infonnation about career fairs, job shadowing, and/or internship oppor- 
tunities. 

_ Information about college readiness (e.g., college tours, SAT prepara- 
tion, FAFSA support). 

(Scale: 1 = Never, 2 = Rarely, 3 = Sometimes, 4 = Often, 5 = Always, as need- 
ed) 

Appendix Table B.4 includes the administrator survey measure: “The school had a data 
system that tracked students’ attendance, behavior, and course perfomiance to identify at-risk 
students during the school year.” The value for the construct was calculated by taking an 
average of the survey item responses. (Three items, Cronbach’s alpha = 0.79.) 

During the [current] school year, did your school regularly track student-level data 
on any of the below characteristics in order to identify at-risk students? 

_ Course performance. 

_ Attendance. 

_ Behavior. 

(Scale: 0 =No, 1 = Yes) 

Appendix Table B.4 includes the administrator survey measure: “Average frequency 
teams of teachers reviewed individual student’s data to identify needs and determine interven- 
tions.” The value for the construct was calculated by taking an average of the survey item 
responses and then rescaling to a 0-10 scale. (Four items, Cronbach’s alpha = 0.95.) 

During the past month, how often did teachers engage in the following activities at 
your school? 

_ Teams of teachers reviewed data on individual students’ academic pro- 
gress and/or course performance to identify student needs. 
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_ Teams of teachers reviewed data on individual students’ attendance to 
identify student needs. 

_ Teams of teachers reviewed data on individual students’ behavior to 
identify student needs. 

_ Teams of teachers determined approaches to respond to identified stu- 
dent needs. 

(Scale: 1 = Never, 2 = At least once a month, 3 = More than once a month, but 
not weekly, 4 = At least once a week, 5 = More than once a week, but not daily, 

6 = Daily) 


Appendix Table B.4 includes the administrator survey measure: “Average frequency 
students participated in individual, group, or family counseling.” The value for the construct 
was calculated by taking an average of the survey item responses and then rescaling to a 0-10 
scale. (Three items, Cronbach’s alpha = 0.75.) 


During the [current] school year, how often were students offered opportunities to 
participate in each of the following activities? 

_ Individual counseling. 

_ Group counseling. 

_ Family counseling. 

(Scale: 1 = Rarely, 2 = Monthly, 3 = Bi-weekly, 4 = Weekly, 5 = Daily) 


Appendix Table B.4 includes the administrator survey measure: “Average frequency 
students were provided with non-academic basic needs when necessary.” The value for the 
construct was calculated by taking an average of the survey item responses and then rescaling to 
a 0-10 scale. (Three items, Cronbach’s alpha = 0.84.) 

During the [current] school year, how frequently were the following supports 
provided to students at your school? 

_ Students were provided with clothing, school supplies, and/or food as 
needed. 

_ Students were provided with health and wellness services as needed. 

_ Students were provided with mental health services as needed. 

(Scale: 1 = Never, 2 = Rarely, 3 = Sometimes, 4 = Often, 5 = Always, when 
needed) 
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Appendix Table B.5 includes the administrator survey measure: “The school had a 
math and English/language arts coach.” The value for the construct was calculated by reversing 
the values for each of the survey items, taking an average of the survey item responses, and then 
rescaling to a 0-10 scale. (Two items, Cronbach’s alpha = 0.85.) 

During the [current] school year, did your school have a Language Arts coach to 
provide teachers with instructional mentoring and support that was. . . 

During the [current] school year, did your school have a Math coach to provide 
teachers with instructional mentoring and support that was. . . 

(Scale: 1 = Working full-time in only your school?, 2 = Working at least half- 
time in your school?, 3 = Working less than half-time in your school?, 4 = Or 
did your school not have one?) 

Appendix Table B.5 includes the administrator survey measure: “Average times per 
month, City Year AmeriCorps members, college students, or volunteers from organized 
programs worked with students.” The value for the construct was calculated by weighting the 
frequency per month and then summing the weighted survey items in order to get a total times- 
per-month measure. For example, 0 = never, 1 = at least once a month, 2 = more than once a 
month but not weekly, 5 = once a week, 12 = more than once a week but not daily, 20 = daily 
(three items). 

During the past month, how often did each of the following volunteers work with 
students in at least some of your classes? 

_ Students from local college/teacher preparation programs. 

_ City Year corps members. 

_ Members of organized volunteer programs or other City Year Ameri- 
Corps members. 

(Scale: 1 = Never, 2 = At least once a month, 3 = More than once a month, but 
not weekly, 4 = Once a week, 5 = More than once a week, but not daily, 6 = 

Daily) 


Methodology for Service Contrast Analysis 

This section describes the statistical model used to estimate the service contrast measures in this 
report. Recall that the service contrast investigates the extent to which the Diplomas Now 
intervention — as implemented in the DN schools — differs from the school practices or other 
reform models implemented in the non-DN schools. The analysis of service contrast is based on 
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teacher and administrator surveys. Since schools were randomly assigned to implement the DN 
model, service contrast measures can be estimated by comparing the average survey responses 
of teachers and administrators in DN and non-DN schools. For each service contrast measure, 
the analysis uses a two-level fixed-effects model, which combines models at the teacher (or 
administrator) and school levels. This approach mimics the one planned for the confirmatory 
impact analyses that will be presented as part of the third report. 

Level 1: Teachers and Administrators Within Schools 

Level 1 describes the relationship between teacher and administrator measures and their 
background characteristics. The Level 1 model is given by: 

Yij ~~ n 0 j + 2^1 s^sij T Yi^-s^sij T &iji (1) 


where 


Y l j is a measure for teacher and administrator i in school /; 

X t j is a set of .S teacher-level and administrator-level covariates for teacher and adminis- 
trator i in school j, centered on the grand mean in the sample; 

M si j is a set of S missing indicators for each of the teacher- level and administrator-level 
characteristics, coded 1 if missing and 0 otherwise; 3 and 

ejj is a random error term for teacher and administrator i from school j, assumed to be 
independently and identically distributed across teachers and administrators within 
schools (that is, the “within-school” residual). 

Therefore, 

7T 0 j is the average of measure Y at school j for the “average” teacher and administrator 
in the sample (that is, with mean value on the S covariates). 

Level 2: Schools 

Given that random assignment occurs at the school level, program impacts are estimat- 
ed at the school level. Thus, Level 2 examines the difference between the school-level adjusted 
measures (7r 0/ ) of DN and non-DN schools, controlling for school characteristics and random 


3 As discussed below, missing information on the teacher and administrator characteristics will be imputed 
using a dummy-variable approach. 
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assignment blocks, where blocks are defined by the district and school level. Therefore: 


n 0 i ~ T.K^kDk + YtKPlkJjD k + u oi 


( 2 ) 


where 


Tj — 1 if school j was randomly assigned to implement the Diplomas Now program and 

0 otherwise; 

D k denotes random assignment block indicators, equal to 1 if teacher and administrator 

1 is in random assignment block k (defined by district and school level) and 0 otherwise; 

u oi is a random error term for school j, assumed to be independently and identically dis- 
tributed across schools (that is, the “between-school” residual). 


Therefore: 

/? lfe is the difference between the school-level average of measure Y in the DN schools 
and the non-DN schools in block/district k, (that is, the impact of the DN program on 
service contrast measure Y in block/district k). 

The two-level model can be estimated by substituting (2) into (1): 

Y ij — T l K^k D k + Y.KPlk T j D k + T.S n ls X sij + u 0i + e ij (3) 

and then fitting Equation (3). 

There are several features to note about these models: 

• The average (pooled) estimate. The average impact of the DN program 
across school district-school level blocks will be obtained by 
weighting the block-level impacts (P ik ) by the number of DN schools in 
the block. Thus, is a fixed-effects estimate of the impact of the Di- 
plomas Now program for the average DN school in the study sample. 
Therefore, the average estimate cannot be used to make statistical infer- 
ences about the impact of the program in some larger population of 
schools. This “fixed-effects” approach to obtaining a pooled impact es- 
timate is used because the school districts in the study were selected pur- 
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posefully and are not a random sample of districts from a larger target 
population. 4 

• Random assignment blocks. Indicators for random assignment blocks 
(D k ) are included in the model to capture a central feature of the research 
design in which random assignment was conducted separately for each 
school district and subgroups of schools within district (that is, middle 
schools and high schools). These blocks also account for all variation in 
mean measure levels across blocks. Thus, the only two sources of varia- 
tion in this model are (1) between schools within blocks and (2) between 
teachers and administrators within schools. 

• Teacher-level and administrator-level co variates. The model allows 
for the inclusion of teacher-level and administrator-level covariates (X i j~) 
to reduce both within- and between-school variation in the service con- 
trast measure, thereby increasing the precision of the impact estimates. 
The decision about which covariates to include in the model was made 
based on whether the covariate was an important predictor of service 
contrast measures in theory. The following teacher-level covariates are 
included in the statistical model: whether a teacher has four years of 
teaching experience or less; whether a teacher has temporary, emergen- 
cy, or provisional certification; whether a teacher has advanced or Na- 
tional Board certification; and whether a teacher has a master’s degree or 
higher level of education. The following administrator-level covariate is 
included in the statistical model: whether an administrator has four years 
of administrator experience or less. When data are missing for covariates, 
and when there are not many covariates with missing values relative to 
the number of observations available, the analysis adopts the “dummy 
variable” approach. This consists of imputing missing values to a given 
variable using the grand mean for the sample, and then including an indi- 


4 Another option for pooling the results would be to use a “random-effects” approach (with block random 
effects), which attempts to estimate the impact of the program for the broader population of sites represented 
by the study sample (as opposed to the fixed-effects approach, which restricts its inferences to the sites in the 
study sample). To date, given the typically small number of sites (districts) for most social experiments, it has 
been common practice to use fixed-effect models for pooling experimental findings (Schochet, 2005). In this 
study, for example, the small number of districts/blocks will not provide enough information about how true 
impacts vary across districts to support generalizations with adequate precision. 
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cator of “missingness” for this variable in statistical models that include 
the variable in question as a co variate. 5 


Ceiling-Effect Analysis for Administrator Surveys 

It is possible that fewer administrator survey service contrast measures showed significant 
differences between DN and non-DN schools than teacher survey service contrast measures 
because of a ceiling effect on administrator responses. In other words, some of the administrator 
items may have had such positive responses across all administrators that they were not success- 
ful in capturing variation, and thus there was no room for a significant difference to emerge 
between DN and non-DN schools. For example, there were eight administrator service contrast 
measures in which survey respondents could choose among at least five different possible 
responses and in which the average response for both DN and non-DN administrators was 
greater than 7.5 on a scale from 0 to 10. For seven of these eight administrator measures there 
was no statistically significant difference between DN and non-DN administrator responses. 
Conversely, not a single teacher measure on the 0-to-10 scale had an average score over 7.5. 

In order to test whether ceiling effects exist, a descriptive analysis was conducted that 
examined the percentage of respondents who got the maximum score on the survey scale, the 
skewness of the distribution, and a histogram of the survey measure. The analysis was only 
conducted for the seven administrator measures for which no significant differences were found 
between DN and non-DN schools. For each of the measures at least 39.7 percent of survey 
respondents gave the maximum score, and each measure was also negatively skewed with a 
skewness statistic of -0.44 or lower. 6 In addition, the histograms of the measures appear to get 
“bunched up” at the top end of the scores. These results suggest that the seven administrator 
measures show signs of having ceiling effects, which, in turn, may partially explain why there 
were limited differences between DN and non-DN schools on these measures. 


Puma. Olsen, Bell, and Price (2009) and internal studies at MDRC have shown that with low rates of 
missing data, this approach yields unbiased estimates of program impacts, and is just as effective as more 
complex imputation methods (for example, regression imputation, expectation-maximization algorithms, etc.). 

6 If the data had ceiling effects, there should be negative values for skewness. Bulmer (1979) suggests this 
rule of thumb regarding how to interpret the magnitude of the skewness statistic: if skewness is less than -1 or 
greater than +1 then the distribution is highly skewed, if skewness is between -1 and -0.5 or between +0.5 and 
+1, the distribution is moderately skewed, if skewness is between -0.5 and +0.5, the distribution is approxi- 
mately symmetric. 
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Service Contrast Supplemental Tables 

This section includes supplementary findings for the service contrast analysis discussed in 
Chapters 2 and 3. Appendix Tables B.2 through B.5 show the pillar-by-pillar Year 2 service 
contrast results for administrator survey measures, which provide additional information about 
differences in school practices between DN and non-DN schools during the second year of 
implementation, the subject of Chapter 2. As part of the comparison of service contrast 
measures across implementation years in Chapter 3, Appendix Tables B.6 through B.9 present 
the Year 1 service contrast findings based on surveys of teachers who taught the sixth and ninth 
grades during the first year of implementation. To reveal whether there were differences in 
service contrast across the whole school compared with the transition years in the second year 
of implementation, Appendix Tables B.10 through B.13 show the Year 2 service contrast 
findings from surveys of teachers who taught the sixth and ninth grades. 
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Diplomas Now 
Appendix Table B.2 


Pillar I: Teacher Teams and Small Learning Communities, 
Service Contrast, Administrator Responses, Year 2 


Survey Item 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Difference 

Effect 

Size 

P-Value 

Strong Learning Environments 






The school had a collective mission statement with 
clearly stated goals and priorities. 

(0 = strongly disagree, 10 = strongly agree) 

7.8 

8.0 

-0.2 

-0.07 

0.632 

The school was subdivided into distinct 
organizational units, such as small learning 
communities. (%) 

58.5 

55.0 

3.5 

0.07 

0.689 

High school administrators reported classes were 
organized in a 4x4 block schedule with extended 
learning periods. (%) 

48.5 

26.7 

21.8 

0.45 

*** 0.001 

Frequency with which interdisciplinary teams of 
teachers engaged in common planning. (0 = never, 
5 = less than weekly, 10 = daily) 

5.4 

4.0 

1.4 

0.48 

** 0.016 

Frequency with which same-subject professional 
learning communities engaged in common planning. 
(0 = never, 5 = less than weekly, 10 = daily) 

5.7 

5.6 

0.0 

0.01 

0.964 

Sample size 

31 

30 





SOURCE: Follow-up surveys of administrators (principals and assistant principals) administered during 
the school years of 2012-2013 and 2013-2014. 


NOTES: Across 61 study schools, 170 administrators participated in the follow-up survey. Across 29 
study high schools, 102 administrators participated in the follow-up survey. One DN school closed prior 
to the second year of implementation and is not included in this analysis. For each of the above 
measures, data are missing for no more than 5.3 percent of the administrators. The difference in the 
percentage of missing data between DN and non-DN schools is no more than 4.3 percent for any of the 
above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school administrators for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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Diplomas Now 
Appendix Table B.3 


Pillar II: Curriculum and Instruction with Professional Development, 
Service Contrast, Administrator Responses, Year 2 


Survey Item 

DN Non-DN 
Schools Schools 

Estimated Effect 
Difference Size 

P -Value 

Professional Development and Peer Coachine 






Average frequency of opportunities to participate 
in a principal/leader support network or other 
professional development. (0 = never, 5 = weekly, 
10 = daily) 

4.2 

4.3 

-0.1 

-0.06 

0.754 

Curriculum for College Readiness 






Curriculum included a course in effective 
transitions to middle or high school. (%) 

68.2 

54.4 

13.9 

0.28 

0.109 

All high school students participated in the same 
college preparation curricula in core academic classes. (%) 

85.1 

80.1 

5.0 

0.13 

0.623 

Average frequency students were provided with 
information about career fairs, job shadowing, 
internship opportunities, or college readiness. 
(high school only) (0 = never, 5 = sometimes, 
10 = always, when needed) 

8.4 

8.2 

0.1 

0.09 

0.776 

Sample size 

31 

30 





SOURCE: Follow-up surveys of administrators (principals and assistant principals) administered during 
the school years of 2012-2013 and 2013-2014. 


NOTES: Across 61 study schools, 170 administrators participated in the follow-up survey. Across 29 
study high schools, 102 administrators participated in the follow-up survey. One DN school closed prior 
to the second year of implementation and is not included in this analysis. For each of the above measures, 
data are missing for no more than 5 percent of the administrators. The difference in the percentage of 
missing data between DN and non-DN schools is no more than 2.7 percent for any of the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school administrators for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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Diplomas Now 
Appendix Table B.4 


Pillar III: Tiered Student Supports, 

Service Contrast, Administrator Responses, Year 2 


Survey Item 

DN 

Schools 

Non-DN 

Schools 

Estimated Effect 
Difference Size 

P- Value 

Tiered Intervention Model 






The school had a data system that tracked students' 
attendance, behavior, and course performance to 
identify at-risk students during the school year. (%) 

98.6 

92.2 

6.4 

0.41 

0.222 

Average frequency teams of teachers reviewed 
individual students' data to identify needs and 
determine interventions. (0 = never, 

5 = less than weekly, 10 = daily) 

5.6 

4.4 

1.1 

0.51 *** 

0.007 

Frequency parents/guardians were invited to meet 
with teachers to discuss strategies for meeting 
individual students' needs. (0 = never, 5 = sometimes, 
10 = always, when needed) 

7.8 

8.2 

-0.4 

-0.19 

0.341 

Student Supports 






Frequency of academic enrichment opportunities 
offered to students outside of normal school hours. 3 
(0 = never, 5 = less than weekly, 10 = daily) 

8.7 

7.6 

1.1 

0.38 * 

0.067 

Frequency students were engaged in mentoring 
services from college students or recent graduates 
who served as role models. (0 = rarely, 5 = biweekly, 
10 = daily) 

3.8 

3.8 

0.0 

0.01 

0.960 

Frequency students were offered opportunities to 
participate in academic tutoring. (0 = rarely, 

5 = biweekly, 10 = daily) 

9.1 

8.6 

0.4 

0.19 

0.339 

Frequency teachers, other staff members, or 
volunteers at the school followed up with students 
when they had attendance issues. (0 = never, 

5 = sometimes, 10 = always, as needed) 

8.0 

7.9 

0.1 

0.08 

0.684 

Students with frequent disruptive behavior received 
a daily check-in with an adult to monitor 
their progress. (%) 

84.3 

75.3 

9.0 

0.20 

0.264 


(continued) 
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Appendix Table B.4 (continued) 


Survey Item 

DN 

Schools 

Non-DN 

Schools 

Estimated Effect 
Difference Size 

P -Value 

Student Case Management 






Average frequency students participated in 
individual, group, or family counseling. (0 = rarely, 
5 = biweekly, 10 = daily) 

7.3 

5.7 

1.6 

0.60 

*** 0.001 

Average frequency students were provided with 

nonacademic basic needs when necessary. 11 

(0 = never, 5 = sometimes, 10 = always, when needed) 

8.2 

8.1 

0.1 

0.07 

0.750 

Students identified as at risk were provided with an 
individual case plan to set goals and monitor 
progress. (%) 

83.9 

80.0 

3.9 

0.09 

0.591 

Sample size 

31 

30 





SOURCE: Follow-up surveys of administrators (principals and assistant principals) administered during 
the school years of 2012-2013 and 2013-2014. 

NOTES: Across 61 study schools, 170 administrators participated in the follow-up survey. One DN 
school closed prior to the second year of implementation and is not included in this analysis. For each of 
the above measures, data are missing for no more than 8.2 percent of the administrators. The difference in 
the percentage of missing data between DN and non-DN schools is no more than 6.7 percent for any of 
the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school administrators for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Academic enrichment opportunities include homework help, service learning tutoring, and test 
preparation. 

b Nonacademic basic needs refer to clothing, school supplies, food, health and wellness services, and 
mental health services. 
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Diplomas Now 


Appendix Table B.5 

Pillar IV: Can-Do Culture and Climate, 
Service Contrast, Administrator Responses, Year 2 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P-Value 

Integrated On-Site Support 






An adult other than the principal or assistant 
principal coordinated school reform efforts. (%) 

64.3 

51.7 

12.6 

0.25 

0.198 

An adult other than the principal or assistant 
principal coordinated interventions and 
community resources. (%) 

88.7 

72.1 

16.6 

0.36 

** 0.050 

The school had a math and English/language arts 
coach. (0 = none, 5 = half-time, 10 = full-time) 

7.0 

4.8 

2.2 

0.54 

*** <0.001 

Average times per month City Y ear AmeriCorps 
members, college students, or volunteers from 
organized programs worked with students. 3 

31.1 

11.6 

19.6 

1.42 

*** <0.001 

Family and Community Involvement 






Average frequency staff members engaged 
with parents/guardians about students' academic, 
behavioral, and attendance issues or progress. 

(0 = never, 5 = less than weekly, 10 = daily) 

7.9 

7.9 

0.1 

0.03 

0.883 

Frequency school sponsored events that included 
participation from families or community members. 
(0 = never, 5 = less than weekly, 10 = daily) 

3.1 

2.8 

0.3 

0.20 

0.413 

Frequency students were offered opportunities 
to participate in service-learning or 
community-service projects. (0 = rarely, 

5 = biweekly, 10 = daily) 

4.6 

4.0 

0.6 

0.16 

0.401 

High school students were offered opportunities 
to participate in work-based learning. (%) 

76.0 

90.0 

-14.1 

-0.44 

0.156 

The school offered services or referrals to 
students' parents/guardians (for example, 
English courses). (%) 

79.8 

81.0 

-1.2 

-0.03 

0.882 

Sample size 

31 

30 





(continued) 
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Appendix Table B.5 (continued) 


SOURCE: Follow-up surveys of administrators (principals and assistant principals) administered 
during the school years of2012-2013 and 2013-2014. 

NOTES: Across 61 study schools, 170 administrators participated in the follow-up survey. Across 29 
study high schools, 102 administrators participated in the follow-up survey. One DN school closed 
prior to the second year of implementation and is not included in this analysis. For each of the above 
measures, data are missing for no more than 5.9 percent of the administrators, except for the 
"administrators reported students were offered opportunities to participate in service-learning or 
community-service projects" item, where 10.6 percent of data are missing. The difference in the 
percentage of missing data between DN and non-DN schools is no more than 5.6 percent for any of the 
above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school administrators for 
the respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 

“Survey items in this input have been calculated by weighting the frequency per month and then 
summing the weighted survey items in order to get a total times-per-month measure. For example, 0 = 
never, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 12 = 
more than once a week but not daily, and 20 = daily. 
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Diplomas Now 
Appendix Table B.6 


Pillar I: Teacher Teams and Small Learning Communities, 
Service Contrast, Teacher Responses, Year 1 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P-Value 

Strong Learning Environments 






Core teachers reported how many of their classes 
were 70 to 90 minutes. (0 = none, 5 = few, 10 = most) 

6.2 

6.3 

-0.1 

-0.03 

0.789 

Core teachers reported the frequency they 
collaborated with an interdisciplinary team of 
teachers who shared the same group of students. 
(0 = never, 5 = less than weekly, 10 = daily) 

5.7 

5.2 

0.5 

0.16 * 

0.087 

Core teachers reported hours spent each week 
collaborating with an interdisciplinary team and 
teaching coordinated content across disciplines. 
(0 = none, 5 = 1 to 2, 10 = 3 or more) 

3.7 

3.5 

0.2 

0.07 

0.446 

Math and English/language arts teachers reported 
participating in a professional learning community 
with teachers within the same subject area. 

(0 = never, 5 = less than weekly, 10 = daily) 

5.3 

5.4 

-0.1 

-0.04 

0.677 

Sample size 

32 

30 





SOURCE: Follow-up surveys of teachers administered during the school years of 201 1-2012 and 
2012-2013. Respondents included middle school teachers who taught sixth grade and high school 
teachers who taught ninth grade. 

NOTES: "Core" academic areas are identified as math. English/language arts, sciences, and social 
studies. Across 62 study schools, 1 ,043 core teachers participated in the follow-up survey, including 
685 math and English/language arts teachers. For each of the above measures, data are missing for no 
more than 3.2 percent of the teachers. The difference in the percentage of missing data between DN 
and non-DN schools is no more than 1 percent for any of the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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Appendix Table B.7 


Pillar II: Curriculum and Instruction with Professional Development, 
Service Contrast, Teacher Responses, Year 1 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P-Value 

Professional Development and Peer Coaching 

Average times per month math and English/language 
arts teachers reported working with an 
instructional coach." 1 

7.1 

6.1 

1.0 0.11 

0.301 

Average times per month math and English/language 
arts teachers reported receiving support from 
a school leader or a coach. a 

13.2 

11.1 

2.1 0.15 

0.224 

Curriculum for College Readiness 

Math and English/language arts teachers reported 
implementing the same college preparation 
curriculum with all students in their classes. 

(high school only) (%) 

55.4 

59.3 

-3.9 -0.08 

0.583 

Math and English/language arts teachers reported 
struggling students in their courses received 
additional classes to catch up with their peers. (%) 

76.4 

81.5 

-5.0 -0.13 

0.205 

Math and English/language arts teachers reported the 
average frequency they used a variety of 
student-centered teaching strategies.* 3 
(0 = never, 5 = less than weekly, 10 = daily) 

0.7 

0.7 

0.0 0.03 

0.782 

Math and English/language arts teachers reported the 
average frequency students applied classroom 
activities to real-life issues and used critical thinking 
and reasoning skills. (0 = never, 5 = less than weekly, 
10 = daily) 

0.7 

0.7 

0.0 0.06 

0.552 

Math and English/language arts teachers reported the 
average frequency academic courses included career 
applications and exploration, (high school only) 

(0 = never, 5 = less than weekly, 10 = daily) 

0.5 

0.5 

0.0 0.02 

0.886 

Sample size 

32 

30 
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Appendix Table B.7 (continued) 


SOURCE: Follow-up surveys of teachers administered during the school years of 2011-2012 and 2012- 
2013. Respondents included middle school teachers who taught sixth grade and high school teachers who 
taught ninth grade. 

NOTES: Across 62 study schools, 685 math and English/language arts teachers participated in the 
follow-up survey. Across 29 study high schools, 404 math and English/language arts teachers 
participated in the follow-up survey. For each of the above measures, data are missing for no more than 
6.4 percent of the teachers. The difference in the percentage of missing data between DN and non-DN 
schools is no more than 1.9 percent for any of the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a Survey items in this input have been calculated by weighting the frequency per month and then 
summing the weighted survey items in order to get a total times-per-month measure. For example, 0 = 
never, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 12 = 
more than once a week but not daily, and 20 = daily. 

b Student-centered teaching strategies include demonstrations, modeling of strategies, and minilessons. 
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Appendix Table B.8 


Pillar III: Tiered Student Supports, 
Service Contrast, Teacher Responses, Year 1 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P-Value 

Tiered Intervention Model 





Core teachers reported the average frequency 
they used attendance, behavior, and course 
performance data to target at-risk students. 

(0 = never, 5 = less than weekly, 10 = daily) 

5.1 

5.4 

-0.3 -0.09 

0.275 

Core teachers reported hours spent each week 
reviewing student data with an interdisciplinary team 
of teachers to identify student needs. (0 = never, 

5 = 1 to 2, 10 = 3 or more) 

3.9 

3.5 

0.4 0.13 

0.110 

Core teachers reported the average frequency 
they participated in meetings with administrators 
or counselors to identify at-risk students and 
plan interventions. (0 = never, 5 = less than weekly, 
10 = daily) 

4.8 

4.1 

0.7 0.24 ** 

0.026 

Core teachers reported the average frequency 
they invited parents of at-risk students to 
participate in discussions about interventions to 
support their child. (0 = never, 5 = less than weekly, 
10 = daily) 

3.9 

4.2 

-0.3 -0.10 

0.388 

Student Supports 





Math and English/language arts teachers reported 
students who were often disruptive were offered 
behavior coaching. (0 = never, 5 = sometimes, 

10 = always, as needed) 

5.6 

5.8 

-0.1 -0.04 

0.703 

Math and English/language arts teachers reported 
the frequency students were provided additional 
support related to attendance. (0 = never, 

5 = less than weekly, 10 = daily) 

6.6 

6.5 

0.1 0.05 

0.641 

Math and English/language arts teachers reported 
the frequency students were offered individual 
or small-group tutoring during their classes. 

(0 = never, 5 = less than weekly, 10 = daily) 

7.5 

7.2 

0.3 0.11 

0.299 


(continued) 
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Appendix Table B.8 (continued) 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P-Value 

Average times per month math and English/ 
language arts teachers reported students received 
whole-class behavioral support from volunteers. 3 

7.0 

-9.5 

16.5 0.22 

* 0.059 

Average times per month math and English/ 
language arts teachers reported students received 
academic help in class from volunteers. 3 

38.3 

26.0 

12.3 0.37 

*** 0.002 

Student Case Management 





Core teachers reported the frequency they arranged 
intensive support and interventions for students 
by making appropriate referrals. (0 = never, 

5 = sometimes, 10 = always, as needed) 

6.9 

7.0 

-0.1 -0.04 

0.657 

Core teachers reported the frequency needs 
assessments were conducted to quickly identify 
students' needs and address them effectively. 

(0 = never, 5 = sometimes, 10 = always, as needed) 

6.1 

5.9 

0.2 0.06 

0.579 

Core teachers reported the frequency academic 
and nonacademic services were coordinated for 
students at risk of dropping out of school. 

(0 = never, 5 = sometimes, 10 = always, as needed) 

6.1 

5.9 

0.2 0.08 

0.419 

Sample size 

32 

30 




SOURCE: Follow-up surveys of teachers administered during the school years of 201 1-2012 and 
2012-2013. Respondents included middle school teachers who taught sixth grade and high school 
teachers who taught ninth grade. 

NOTES: "Core" academic areas are identified as math. English/language arts, sciences, and social 
studies. Across 62 study schools, 1,043 core teachers participated in the follow-up survey, including 
685 math and English/language arts teachers. For each of the above measures, data are missing for no 
more than 6.9 percent of the teachers. The difference in the percentage of missing data between DN 
and non-DN schools is no more than 2 percent for any of the above measures. 

A two-tailed t-test is used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 

“Survey items in this input have been calculated by weighting the frequency per month and then 
summing the weighted survey items in order to get a total times-per-month measure. For example, 0 = 
never, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 12 = 
more than once a week but not dailv. and 20 = dailv. 
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Appendix Table B.9 


Pillar IV: Can-Do Culture and Climate, 
Service Contrast, Teacher Responses, Year 1 


Survey Item 

DN 

Schools 

Non-DN Estimated Effect 
Schools Difference Size 

P-Value 

Integrated On-Site Support 






Average times per month math and English/language 
arts teachers reported City Year AmeriCorps members 
worked with students. 3 

12.9 

3.4 

9.5 

1.40 

*** <0.001 

Average times per month math and English/language 
arts teachers reported City Year AmeriCorps members, 
college students, or volunteers from organized 
programs worked with students. 3 

16.3 

5.5 

10.8 

1.07 

*** <0.001 

Average times per month math and English/language 
arts teachers reported any volunteer worked 
with students. 3 ' 13 

21.7 

10.0 

11.7 

0.66 

*** <0.001 

Family and Communitv Involvement 






Teachers reported the frequency parents and 
community members were offered opportunities to 
participate in school initiatives. (0 = never, 5 = less 
than weekly, 10 = daily) 

0.4 

0.4 

0.0 

-0.01 

0.937 

Teachers reported the school had a plan for parent 
and community engagement linked to specific goals 
for improving student learning and healthy 
development. (%) 

49.2 

51.3 

-2.1 

-0.04 

0.677 

Teachers reported the school helped all parents 
understand what they could do at home to support 
a student's success in school. 

(0 = strongly disagree, 10 = strongly agree) 

0.6 

0.6 

0.0 

0.01 

0.937 

Sample size 

32 

30 
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Appendix Table B.9 (continued) 


SOURCE: Follow-up surveys of teachers administered during the school years of 201 1-2012 and 2012- 
2013. Respondents included middle school teachers who taught sixth grade and high school teachers who 
taught ninth grade. 

NOTES: Across 62 study schools, 1,620 teachers participated in the follow-up survey including 685 math 
and English/language arts teachers. For each of the above measures, data are missing for no more than 
8.6 percent of the teachers. The difference in the percentage of missing data between DN and non-DN 
schools is no more than 1 .8 percent for any of the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 

“Survey items in this input have been calculated by weighting the frequency per month and then 
summing the weighted survey items in order to get a total times-per-month measure. For example, 0 = 
never, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 1 2 = more 
than once a week but not daily, and 20 = daily. 

b Volunteers include: City Year AmeriCorps members, students from local college or 
teacher-preparation programs, members of organized volunteer groups, peer tutors, and community 
members. 
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Appendix Table B.10 

Pillar I: Teacher Teams and Small Learning Communities, 
Service Contrast, Teacher Responses, Sixth and Ninth Grades, Year 2 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P- Value 

Strong Learning Environments 






Core teachers reported how many of their classes 
were 70 to 90 minutes. (0 = none, 5 = few, 10 = most) 

6.5 

5.1 

1.5 

0.31 *** 

<0.001 

Core teachers reported the frequency they 
collaborated with an interdisciplinary team of 
teachers who shared the same group of students. 
(0 = never, 5 = less than weekly, 10 = daily) 

5.7 

4.3 

1.5 

0.45 *** 

0.002 

Core teachers reported hours spent each week 
collaborating with an interdisciplinary team and 
teaching coordinated content across disciplines. 
(0 = none, 5 = 1 to 2, 10 = 3 or more) 

4.1 

3.2 

0.9 

0.33 *** 

0.003 

Math and English/language arts teachers reported 
participating in a professional learning community 
with teachers within the same subject area. 

(0 = never, 5 = less than weekly, 10 = daily) 

5.5 

5.1 

0.4 

0.16 

0.255 

Sample size 

30 

30 





SOURCE: Follow-up surveys of teachers administered during the school years of 2012-2013 and 
2013-2014. Respondents included middle school teachers who taught sixth grade and high school 
teachers who taught ninth grade. 

NOTES: "Core" academic areas are identified as math. English/language arts, sciences, and social 
studies. Across 60 study schools, 853 core teachers participated in the follow-up survey, including 577 
math and English/language arts teachers. Two study schools were excluded from the analysis due to 
missing survey data. For each of the above measures, data are missing for no more than 3.3 percent of 
the teachers. The difference in the percentage of missing data between DN and non-DN schools is no 
more than 1 percent for any of the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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Appendix Table B.ll 


Pillar II: Curriculum and Instruction with Professional Development, 
Service Contrast, Teacher Responses, Sixth and Ninth Grades, Year 2 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P-Value 

Professional Development and Peer Coaching 

Average times per month math and English/language 
arts teachers reported working with an 
instructional coach.' 1 

8.5 

4.9 

3.7 0.41 ** 

0.013 

Average times per month math and English/language 
arts teachers reported receiving support from 
a school leader or a coach. 3 

15.0 

9.2 

5.9 0.41 ** 

0.010 

Curriculum for College Readiness 

Math and English/language arts teachers reported 
implementing the same college preparation 
curriculum with all students in their classes. 

(high school only) (%) 

60.2 

60.3 

-0.2 0.00 

0.983 

Math and English/language arts teachers reported 
struggling students in their courses received 
additional classes to catch up with their peers. (%) 

71.3 

72.2 

-1.0 -0.02 

0.877 

Math and English/language arts teachers reported the 
average frequency they used a variety of 
student-centered teaching strategies. 13 
(0 = never, 5 = less than weekly, 10 = daily) 

0.7 

0.7 

0.0 0.14 

0.194 

Math and English/language arts teachers reported the 
average frequency students applied classroom 
activities to real-life issues and used critical thinking 
and reasoning skills. (0 = never, 5 = less than weekly, 
10 = daily) 

0.7 

0.7 

0.0 0.06 

0.558 

Math and English/language arts teachers reported the 
average frequency academic courses included career 
applications and exploration, (high school only) 

(0 = never, 5 = less than weekly, 10 = daily) 

0.5 

0.5 

0.0 0.04 

0.744 

Sample size 

30 

30 
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Appendix Table B.ll (continued) 


SOURCE: Follow-up surveys of teachers administered during the school years of 2012-2013 and 2013- 
2014. Respondents included middle school teachers who taught sixth grade and high school teachers who 
taught ninth grade. 

NOTES: Across 60 study schools, 577 math and English/language arts teachers participated in the 
follow-up survey. Across 29 study high schools, 332 math and English/language arts teachers 
participated in the follow-up survey. Two study schools were excluded from the analysis due to missing 
survey data. For each of the above measures, data are missing for no more than 6.6 percent of the 
teachers. The difference in the percentage of missing data between DN and non-DN schools is no more 
than 3.4 percent for any of the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Survey items in this input have been calculated by weighting the frequency per month and then 
summing the weighted survey items in order to get a total times-per-month measure. For example, 0 = 
never, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 12 = 
more than once a week but not daily, and 20 = daily. 

b Student-centered teaching strategies include demonstrations, modeling of strategies, and minilessons. 
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Appendix Table B.12 

Pillar III: Tiered Student Supports, 

Service Contrast, Teacher Responses, Sixth and Ninth Grades, Year 2 


Survey Item 

DN Non-DN Estimated Effect 
Schools Schools Difference Size 

P- Value 

Tiered Intervention Model 






Core teachers reported the average frequency 
they used attendance, behavior, and course 
performance data to target at-risk students. 

(0 = never, 5 = less than weekly, 10 = daily) 

5.6 

4.7 

0.8 

q 29 *** 

0.006 

Core teachers reported hours spent each week 
reviewing student data with an interdisciplinary team 
of teachers to identify student needs. (0 = never, 

5 = 1 to 2, 10 = 3 or more) 

4.2 

3.3 

0.9 

0.33 *** 

0.006 

Core teachers reported the average frequency 
they participated in meetings with administrators 
or counselors to identify at-risk students and 
plan interventions. (0 = never, 5 = less than weekly, 
10 = daily) 

5.0 

3.4 

1.6 

0.56 *** 

<0.001 

Core teachers reported the average frequency 
they invited parents of at-risk students to 
participate in discussions about interventions to 
support their child. (0 = never, 5 = less than weekly, 
10 = daily) 

4.3 

4.1 

0.2 

0.08 

0.361 

Student Supports 






Math and English/language arts teachers reported 
students who were often disruptive were offered 
behavior coaching. (0 = never, 5 = sometimes, 

10 = always, as needed) 

5.7 

4.9 

0.8 

0.26 ** 

0.018 

Math and English/language arts teachers reported 
the frequency students were provided additional 
support related to attendance. (0 = never, 

5 = less than weekly, 10 = daily) 

6.9 

6.2 

0.7 

0.22 ** 

0.041 

Math and English/language arts teachers reported 
the frequency students were offered individual 
or small-group tutoring during their classes. 

(0 = never, 5 = less than weekly, 10 = daily) 

7.3 

6.8 

0.5 

0.18 * 

0.100 
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Appendix Table B.12 (continued) 


Survey Item 


DN Non-DN Estimated Effect 
Schools Schools Difference Size P-Value 


Average times per month math and English/ 
language arts teachers reported students received 
whole-class behavioral support from volunteers. 3 

Average times per month math and English/ 
language arts teachers reported students received 
academic help in class from volunteers. 3 

Student Case Management 

Core teachers reported the frequency they arranged 
intensive support and interventions for students 
by making appropriate referrals. (0 = never, 

5 = sometimes, 10 = always, as needed) 

Core teachers reported the frequency needs 
assessments were conducted to quickly identify 
students' needs and address them effectively. 

(0 = never, 5 = sometimes, 10 = always, as needed) 

Core teachers reported the frequency academic 
and nonacademic services were coordinated for 
students at risk of dropping out of school. 

(0 = never, 5 = sometimes, 10 = always, as needed) 

Sample size 


8.0 4.1 3.9 0.54 *** 0.001 


46.5 22.2 24.3 0.73 *** <0.001 


7.2 7.1 


6.1 6.1 


6.4 6.3 

"30 30 


0.1 0.21 ** 0.034 


0.1 0.22 ** 0.016 


0.1 0.42 *** <0.001 


SOURCE: Follow-up surveys of teachers administered during the school years of 2012-2013 and 
2013-2014. Respondents included middle school teachers who taught sixth grade and high school 
teachers who taught ninth grade. 

NOTES: "Core" academic areas are identified as math. English/language arts, sciences, and social 
studies. Across 60 study schools, 853 core teachers participated in the follow-up survey, including 577 
math and English/language arts teachers. Two study schools were excluded from the analysis due to 
missing survey data. For each of the above measures, data are missing for no more than 7.6 percent of 
the teachers. The difference in the percentage of missing data between DN and non-DN schools is no 
more than 6.9 percent for any of the above measures. 

A two-tailed t-test is used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Survey items in this input have been calculated by weighting the frequency per month and then 
summing the weighted survey items in order to get a total times-per-month measure. For example, 0 = 
never, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 12 = 
more than once a week but not daily, and 20 = daily. 
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Appendix Table B.13 

Pillar IV: Can-Do Culture and Climate, 

Service Contrast, Teacher Responses, Sixth and Ninth Grades, Year 2 


Survey Item 

DN 

Schools 

Non-DN Estimated Effect 
Schools Difference Size 

P-Value 

Integrated On-Site Support 






Average times per month math and English/language 
arts teachers reported City Year AmeriCorps members 
worked with students. 3 

14.4 

1.6 

12.8 

2.07 

*** <0.001 

Average times per month math and English/language 
arts teachers reported City Year AmeriCorps members, 
college students, or volunteers from organized 
programs worked with students. 3 

18.9 

3.6 

15.3 

1.68 

*** <0.001 

Average times per month math and English/language 
arts teachers reported any volunteer worked 
with students. a,b 

26.0 

7.7 

18.3 

1.02 

*** <0.001 

Family and Community Involvement 






Teachers reported the frequency parents and 
community members were offered opportunities to 
participate in school initiatives. (0 = never, 5 = less 
than weekly, 10 = daily) 

0.4 

0.4 

0.0 

0.07 

0.426 

Teachers reported the school had a plan for parent 
and community engagement linked to specific goals 
for improving student learning and healthy 
development. (%) 

55.8 

55.8 

0.0 

0.02 

0.838 

Teachers reported the school helped all parents 
understand what they could do at home to support 
a student's success in school. 

(0 = strongly disagree, 10 = strongly agree) 

0.6 

0.6 

0.0 

0.05 

0.515 

Sample size 

30 

30 
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Appendix Table B.13 (continued) 

SOURCE: Follow-up surveys of teachers administered during the school years of 2012-2013 and 2013- 
2014. Respondents included middle school teachers who taught sixth grade and high school teachers who 
taught ninth grade. 

NOTES: Across 60 study schools, 1,339 teachers participated in the follow-up survey, including 577 
math and English/language arts teachers. Two study schools were excluded from the analysis due to 
missing survey data. For each of the above measures, data are missing for no more than 7.6 percent of the 
teachers. The difference in the percentage of missing data between DN and non-DN schools is no more 
than 3.5 percent for any of the above measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Effect sizes were computed using the standard deviations of all non-DN school teachers for the 
respective measures. 

Rounding may cause slight discrepancies in calculating sums and differences. 

“Survey items in this input have been calculated by weighting the frequency per month and then 
summing the weighted survey items in order to get a total times-per-month measure. For example, 0 = 
never, 1 = at least once a month, 2 = more than once a month but not weekly, 5 = once a week, 12 = more 
than once a week but not daily, and 20 = daily. 

b Volunteers include: City Year AmeriCorps members, students from local college or 
teacher-preparation programs, members of organized volunteer groups, peer tutors, and community 
members. 
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Appendix C 

Qualitative Data from Case Study Sites 




This appendix provides additional details about qualitative case study data-collection activities 
at nine case study schools from four school districts during the second year of Diplomas Now 
implementation. (These findings are primarily discussed in Chapter 4.) The first section in- 
cludes counts of participants in interviews and focus groups (Appendix Table C.l), the second 
section describes the qualitative research methodology, and the final section includes the fidelity 
of implementation findings for the nine schools that participated in the case studies (Appendix 
Table C.2). 


Interview and Focus Group Response Counts 

Appendix Table C.l includes the response counts of interview and focus group participants 
during the spring of the second year of implementation. 


Diplomas Now 
Appendix Table C.l 


Counts of Interview and Focus Group Participants, 
Case Study Schools 



DN Middle Schools 

DN High Schools 

All DN Schools 


Total 

Participants 

Total 

Schools 

Total 

Participants 

Total 

Schools 

Total 

Participants 

Total 

Schools 

Case study round 

Spring 2013 

32 

1 

117 

3 

150 

4 

Spring 2014 

104 

3 

45 

2 

151 

5 

Respondent type 

Diplomas Now staff members 

57 

4 

77 

5 

137 

9 

District/school staff members 

30 

4 

42 

5 

72 

9 

Parents 

15 

3 

10 

4 

25 

7 

Students 

34 

4 

33 

5 

67 

9 

Sample size 

136 

4 

162 

5 

301 

9 


SOURCES: Diplomas Now program staff, school, and district interviews and focus groups, 2013 and 
2014. 

NOTES: Data were collected at nine DN schools representing four districts. The total number of 
participants is higher than the sum of the middle and high school total participants because three 
interviews were conducted with field managers who oversaw both DN middle schools and DN high 
schools, and who were not included in either separate count. 
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Case Study Methodology 

Case study research team members used standardized processes for the collection and analysis 
of case study data to ensure the consistent use of best practices. The following sections provide 
detail regarding interview and focus group protocol development, on-site data-collection 
activities, and qualitative analysis procedures. 

Protocol Development 

Interview protocols, focus group guides, and consent forms used by the case study re- 
search team were preapproved by three separate Institutional Review Boards from ICF Interna- 
tional, Johns Hopkins University, and MDRC to ensure human subject protections during each 
round of case study data collection. 1 Each protocol was developed based on the program logic 
model and the following five research questions to gather in-depth infonnation about partici- 
pants’ experiences during the second year of implementation: 

1. How was the Diplomas Now model implemented in case study schools? 

2. What factors facilitated or hindered implementation of the model? 

3. What are the perceived benefits of the Diplomas Now model in case study 
schools? 

4. What were the drawbacks to implementing the Diplomas Now model? 

5. What lessons can be learned from the implementation of the Diplomas Now 
model in case study schools? 

Interview and focus group protocols contained similar sets of open-ended questions tai- 
lored to each participant group and focused on key topics including: implementation, collabora- 
tive activities, perceived impact, lessons learned, and sustainability. 

Data Collection 

Case study data-collection activities were conducted with four schools in spring 2013 
and five schools in spring 2014 to capture the range of programmatic experiences from school 


'Protocols and focus group guides can be found in the Data Collection Instrument Supplement to the first 
report, Corrin et al. (2014b), available at www.mdrc.org. 
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staff members, program staff members, parents, and students during the second year of imple- 
mentation. Case study schools represented four mostly urban school districts and included four 
middle schools and five high schools. Two to three research team members visited each school 
for three to five days to conduct interviews and facilitate focus groups. Individual interviews — 
approximately 60 minutes in length — included school-based program staff members (for 
example, Talent Development school transformation facilitators, City Year program managers 
and team leaders, Communities In Schools site coordinators, school administrators, school 
counselors, and instructional coaches) and district-based staff members (for example, instruc- 
tional facilitators, Diplomas Now field managers, school and student support services facilita- 
tors, and school district leaders). Focus groups lasted between 60 and 90 minutes and were 
conducted separately with parents, students, teachers, and City Year AmeriCorps members, and 
ranged from 2 to 1 1 participants per group. Interviews and focus groups were digitally recorded 
and supplemented by research team members’ notes. Following each site visit, the research 
team completed site-visit summaries to capture overall impressions and main takeaways from 
the visit. All interviews and focus group recordings were transcribed verbatim. 

Coding Processes and Procedures 

Transcripts were stored, managed, and analyzed using the qualitative data analysis 
software Atlas.ti. This software was selected based on the research team’s previous experience 
using it and because it makes it easy to synthesize information drawn from different participant 
groups and data sources. Qualitative data were segmented into “quotations,” or sections/units of 
data that were relevant and that stood on their own. 2 Not all sections of transcripts were as- 
signed codes and some quotations received multiple or overlapping codes. 

The research team used the same coding framework to analyze Year 2 data that was 
used to analyze Year 1 data. (See Box C.l for the coding framework.) An inductive process of 
coding interview and focus group data was iterative and continuously driven both by research 
purpose and by the data themselves. 3 The research team developed an initial set of codes prior 
to analyzing Year 1 case study data, following a review of the program logic model and other 
program materials. Codes, with their corresponding definitions, were documented in a code- 
book and the research team reviewed the codebook collectively to facilitate a shared under- 
standing prior to coding. 


2 Lincoln and Cuba (1985). 

3 Merriam (2009). 
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Box C.1 


Coding Framework for Interviews and Focus Groups 

Activity type. Program inputs and activities specific to Diplomas Now. 

Background information. Introductory and background information on interview/focus 
group participants. 

Implementation challenges. Activities/practices identified as challenging to implement 
or barriers that reportedly made implementation a challenge. 

Implementation facilitators. Activities/practices identified as effectively implemented or 
factors that reportedly made implementation successful. 

Collaboration. Types of interactions to establish and capitalize on relationships (for ex- 
ample, formal meetings, informal communication, or planning) among key players such as 
Diplomas Now school-based staff members, administrators, and teachers. 

Perceived program impact. Perceived effects that participants attributed to Diplomas 
Now program-specific activities/inputs and details about whom or what was affected (for 
example, student behavior, student attendance, student course performance, or school 
climate). 

Lessons learned. Things respondents would do differently or recommendations from the 
first year of implementation (for example, lessons, quotes, examples, or stories). 

School context. Information about the setting in which Diplomas Now is implemented 
(for example, regarding resources, school characteristics, or staff stability) or external 
factors that may be affecting implementation (for example, regarding school closures, 
principal turnover, or teacher layoffs). 


The research team conducted two phases of coding. The first phase included initial 
content analysis, whereby codes, established a priori, were applied to data . 4 Each transcript 
was independently analyzed by two research team members. To ensure inter-rater reliability, 
the team resolved any discrepant analyses during consensus-building meetings that occurred 
every two weeks. These regular meetings for peer debriefing also served as an opportunity to 
con-finn consistent application of processes, provide feedback on trends in coding (for 
example, on the over- or underemphasis of particular codes), clarify or revise operational 
definitions of codes, organize codes into logical groups, and determine the need for additional 
codes. Coding decisions were discussed and analyzed until the team members reached 
agreement and clarified inconsistencies. In the second phase of coding, research team mem- 


4 Miles and Huberman (1994); Saldana (2009); Patton (2002). 
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bers conducted ad hoc analysis to identify additional themes not previously identified using a 
priori codes. This final review was conducted to ensure the application of ad hoc codes to all 
transcripts. The research team also applied the “memo” function in Atlas.ti to document 
preliminary interpretations for reporting purposes and to document questions to discuss and 
clarify during consensus-building meetings. 

Data Analysis Processes and Procedures 

After completing two phases of coding, the team analyzed the frequency with which 
each code was applied to transcripts, or its “groundedness,” to determine which codes should be 
aggregated, disaggregated, or deleted. In other words, codes with no or few quotations (low 
groundedness) were merged with other codes and codes with high groundedness were further 
analyzed to consider whether it was necessary to break them apart into more specific codes. 
This process looked for correspondence among codes, and helped achieve a manageable 
number of codes based on frequency, uniqueness, and importance. 5 

Code frequency tables and code co-occurrence tables were used to identify prominent 
themes, similarities, and trends from Year 1 to Year 2. Narrative summaries were used as a 
data-reduction technique and included an additional process to group quotations for further 
inquiry based on commonly co-occurring themes. 6 This allowed the research team to fully 
understand emerging themes based on the data and to pull direct quotations or vignettes to 
illustrate relevant concepts. 


Fidelity Scores for Case Study Schools 

Appendix Table C.2 includes the Year 1 and Year 2 average fidelity scores for the nine schools 
that participated in the case study. 


5 Creswell (1998); Merriam (1998). 

6 Seidman (2006); Maxwell (1995). 
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Diplomas Now 
Appendix Table C.2 


Fidelity of Implementation Findings, Case Study Schools, Comparison of Year 1 and Year 2 


Model Inputs 

Average Fidelity Score 
Y ear 1 Y ear 2 

Difference 

Pillar I. Teacher Teams and Small Learning Communities 

0.80 

0.80 

-0.01 

• • ’ j j ") 

Strong Learning Environments ’ 

0.80 

0.80 

-0.01 

Pillar II. Curriculum and Instruction with Professional Development 

0.44 

0.37 

-0.07 

Professional Development and Peer Coaching 3 ' 

0.56 

0.37 

-0.20 

Curriculum for College Readiness 

0.32 

0.37 

0.05 

Pillar III. Tiered Student Supports 

0.73 

0.72 

-0.01 

Tiered Intervention Model 11 

0.81 

0.81 

0.01 

Student Supports 3 

0.72 

0.77 

0.04 

Student Case Management 3 

0.65 

0.57 

-0.08 

Pillar IV. Can-Do Culture and Climate 

0.64 

0.65 

0.00 

Integrated On-Site Support 3 

0.86 

0.85 

-0.02 

Family and Community Involvement 

0.42 

0.45 

0.03 

Program Staff Training and Development 

0.68 

0.66 

-0.01 

Overall 

0.65 

0.63 

-0.02 

Sample size 

7 

9 



(continued) 




Appendix Table C.2 (continued) 

SOURCE: Diplomas Now fidelity of implementation program staff surveys, 2012, 2013, 
and 2014. 

NOTES: Each of the nine inputs consists of a set of more specific components measured 
on a 0-1 scale. For each case study school, all of the component scores under an input are 
averaged to create the implementation score for that input. These scores are then averaged 
to create the pillar-level and overall implementation scores. 

A two tailed t-test was applied to differences. No significant differences were found 
between groups with the exception of Strong Learning Environments in Year 1, for which 
case study schools scores were significantly better than non-case study schools. 

Rounding may cause slight discrepancies in calculating sums and differences. 
“Indicates model inputs designated as critical to the Diplomas Now model. 
b Strong Learning Environments, Professional Development and Peer Coaching, and 
Curriculum for College Readiness include some components specific to either middle 
schools or high schools. 
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Laying Tracks to Graduation 

The First Year of Implementing Diplomas Now 

2014. William Corrin, Susan Sepanik, Aracelis Gray, Felix Fernandez, Ashley Briggs, Kathleen K. 
Wang 

Data Collection Instrument Supplement to Laying Tracks to Graduation 
The First Year of Implementing Diplomas Now 

2014. William Corrin, Susan Sepanik, Aracelis Gray, Felix Fernandez, Ashley Briggs, Kathleen K. 
Wang 


NOTE: All the publications listed above are available for free download at www.mdrc.org. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



